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CHAPTER  I 


INTRODUCTION 


Historically,  Air  Force  policy  has  advocated  the 
use  of  organic  logistics  support  for  weapon  systems  and 
equipment  from  the  date  of  first  article  delivery  to  an 
operational  unit.  Planning  for  such  a  capability  must 
begin  in  the  early  acquisition  phases  of  a  weapon  system's 
life  cycle,  and  requires  early  identification  and 
acquisition  of  the  necessary  spares,  support  equipment, 
technical  documentation,  facilities,  training  equipment, 
and  training.  It  also  requires  the  establishment  of  base 
level  on-  and  off-equipment  maintenance  facilities  and 
depot  level  maintenance  facilities.  As  the  weapon  system 
design  changes  and  matures,  all  the  logistics  requirements 
are  subject  to  change  also.  Premature  acquisition  of  such 
logistics  resources  can  thus  lead  to  costly  modifications 
and/or  obsolescence  of  those  resources.  For  this  reason, 
the  Air  Force  and  the  Department  of  Defense  (DOD)  began 
investigating  possible  alternatives  to  the  use  of  organic 
logistics  support  at  initial  operational  capability  (IOC). 

In  October,  1971,  former  Deputy  Secretary  of  Defense 


David  Packard  stated, 


We  are  making  decisions  to  acquire  an  organic 
logistic  support  capability  for  major  weapon 
systems  far  too  early  in  the  acquisition  process. 
.  .  .  These  decisions  reached  before  the  design 
has  stabilized  results  [sic]  in  compounding  our 
costs  for  this  support  au<=  to  the  necessity  for 
accommodating  the  summation  of  the  thousands  of 
engineering  changes  that  occur  just  prior  to  and 
subsequent  to  test  programs  ...  We  must  assure 
that  such  decisions  are  not  made  until  we 
have  reasonable  assurance  that  the  design  has 
stabilized  to  a  point  where  engineering  changes 
will  not  be  made  that  significantly  impact  on 
our  decisions  to  acquire  a  full-fledged  logistic 
support  capability.  I  can  see  no  reason  why  we 
can't  rely  on  the  contractor  for  such  logistic 
support  prior  to  design  stabilization  [15]. 


Air  Force  and  Department  of  Defense  (DOD)  study  groups 
formed  in  the  early  1970s  reconfirmed  that 


Unrealistic  initial  operational  capability 
schedules  and  the  associated  organic  logistic 
support  dates  often  resulted  in  costly  decisions 
to  prematurely  acquire  support  equipment,  spares, 
and  technical  data  [9:76]. 


They  emphasized  that  the  achievement  of  an  organic  main¬ 
tenance  capability  "must  be  determined  primarily  by  the 
design  stability  of  individual  components  and  subsystems 
[9:76]"  of  prime  mission  equipment  (PME ) . 

A  joint  DOD-industry  integrated  logistic  support 
advisory  committee,  formed  in  1972,  recommended  that 
cost-effectiveness  and  design  stability  "be  given  as  much 
emphasis  as  operational  requirements  [9:76]"  when  making 
decisions  to  use  contractor  initial  support.  DOD  Directive 
4140.40,  "Basic  Objectives  and  Policies  on  Provisioning  of 


End  Items  of  Materiel,"  directs  that  "the  integrated 
logistic  support  plan  (ILSP)  provide  for  the  most 
cost-effective  logistic  support  posture  (24).  It 
identifies  items  of  low  reliability,  unstable  design,  high 
risk,  and/or  high  unit  cost  as  the  most  likely  candidates 
for  ICS.  Such  items  can  be  identified  through  the 
logistics  support  analysis  (LSA)  process. 


LSA  includes  the  determination  and  establishment  of 
logistic  support  design  constraints,  consideration 
of  those  constraints  in  the  design  of  the  hardware 
portion  of  the  system,  and  analysis  of  design  to 
validate  the  logistic  support  feasibility  of  the 
design  and  to  identify  and  document  the  logistic 
support  resources  which  must  be  provided,  as  a  part 
of  the  system/equipment,  to  the  operating  forces 
(AFR  800-7)  [17:400]. 


Review  of  historical  data  from  other,  similar  weapon  system 
programs  can  aid  this  process  by  identifying  those  items 
which  have  been  prone  to  design  stability  or  resource 
availability  problems  (11:2-16). 

These  changes  in  Air  Force  philosophy  regarding 
initial  logistic  support  capability  culminated  in 
September,  1975  with  the  publication  of  Air  Force 
Regulation  (AFR)  800-21,  "Interim  Contractor  Support  for 
Systems  and  Equipment."  AFR  800-21  defines  interim 
contractor  support  (ICS)  as 


a  cost-effective  logistics  support  alternative 
for  a  major  system  or  high  cost  or  risk  Class  V 
Modification.  It  allows  the  Air  Force  to  defer 
investment  in  all  or  part  of  the  support  resources 
(such  as  spares,  technical  data,  support  equipment. 


and  training  equipment)  and  to  use  contractor 
support  while  the  organic  capability  is  being 
phased  in  [21:para  l.a]. 

According  to  AFR  800-21,  ICS  has  a  specific  meaning  and 
refers  to  the  use  of  a  contractor  only  in  the  two 
situations  described  below. 

Situation  1.  The  items  to  be  supported  or  items  of 
support  equipment  have  an  unstable  design;  moreover,  the 
projected  cost  of  setting  up  an  organic  capability  at  the 
time  operational  support  is  first  required  is  excessive, 
either  because  of  uncertainties  in  the  type  and  level 
of  support  required,  or  because  of  the  risk  that  support 
resources  will  become  obsolete  if  procured  too  early. 

For  contractor  support  to  be  described  as  ICS  in  this 
situation,  it  must  have  been  planned  at  least  budget  lead 
time  away  and  have  been  subjected  to  rigorous  cost  and 
risk  analyses. 

Situation  2.  All  or  part  of  the  resources  required 
to  establish  an  organic  capability  will  not  be  available 
until  after  operational  support  is  first  required.  In 
this  situation,  the  system  development,  production,  and 
deployment  phases  do  not  allow  enough  time  to  develop  the 
support  resources  before  organic  support  is  needed.  (The 
most  common  example  of  this  situation  is  the  sequential 
phasing  of  automatic  test  equipment  development,  which 
generally  lags  development  of  the  system  it  supports.)  I: 


Situation  2,  as  in  Situation  1,  to  qualify  as  ICS,  early 
planning  and  analyses  must  occur  (21:paras  l.a.l.a&b). 

Support  objectives  identified  in  APR  800-21  include 
1)  reducing  the  risk  of  incurring  unnecessary  costs  by 
minimizing  the  initial  investment  in  organic  support 
resources  when  parts  of  the  system  being  procured  have 
items  with  unstable  design  characteristics;  and  2)  using 
ICS  where  economic  and  risk  analysis  studies  of  design  and 
logistics  support  alternatives  show  that  this  approach  is 
cost-effective,  especially  for  those  parts  of  the  system 
that  are  complex,  expensive,  or  unstable  (21:paras  2.b&c). 


Under  the  ICS  concept,  the  Air  Force  conducts 
risk  analyses  and  other  evaluations  through  the 
logistics  support  analysis  (LSA)  process,  to 
identify  potentially  unstable  design  items  and 
items  having  potentially  high  support  costs. 
Through  these  analyses,  it  identifies  those  areas 
where  it  may  be  beneficial  to  use  a  contractor 
for  logistics  support  during  the  early  deployment 
phase  [21: para  3. a]. 


Such  risk  analyses  should  be  conducted  during  the  demon¬ 
stration  and  validation  phases  of  a  weapon  system  program 
and  during  full-scale  engineering  development  as  a  part  of 
LSA,  refining  the  list  of  ICS  candidate  items  as  improved 
information  is  generated. 

A  risk  analysis  is  defined  in  AFR  800-21  as 

an  analysis  that  evaluates  expected  performance 
(cost,  schedule,  or  technical)  as  compared  with 
desired  performance,  with  a  view  toward  determining 
the  probability  that  requirements  will  be  met 
within  available  resources.  This  includes  iden¬ 
tifying  the  areas  of  uncertainty,  assessing  the 


probabilities,  analyzing  to  determine  the  driving 
or  dominant  parameters,  evaluating  funding  alter¬ 
natives,  making  tradeoff  studies,  and  making 
decisions  on  course(s)  of  action  [21:para  l.h]. 


An  unstable  design  is  defined  in  AFR  800-21  as 

a  design  that  has  a  high  potential  for  change  and 
may  require  more  engineering  in  order  to  meet  the 
design  specification  requirements  for  operational 
performance,  producibility,  maintainability,  or 
reliability.  Design  stability  considerations 
include  determination  or  confirmation  of  the 
equipment  failure  mode  and  effects  pattern  under 
normal  operating  circumstances  [19: para  l.k]. 


ICS  Planning 


A  critical  element  in  the  definition  of  ICS  is  the 
requirement  that  it  be  planned  for  at  least  budget  lead 
time  away.  The  budget  cycle  takes  a  minimum  of  two  years 
from  the  start  of  the  planning  phase  to  the  beginning  of 
the  fiscal  year,  and  budget  cycles  overlap  for  the  various 
fiscal  years.  The  Air  Force  portion  of  the  DOD  Five-Year 
Defense  Program  (FYDP ) ,  the  Force  and  Financial  Program, 
requires  five-year  cost  projections  be  made  at  the 
beginning  of  the  budget  cycle  for  each  fiscal  year. 

AFLC  is  responsible  for  logistic  support,  and  therefore 
is  responsible  for  budgeting  and  funding  for  all  ICS 
requirements.  ICS  is  funded  on  a  fiscal  year  basis,  and 
requirements  must  be  projected  and  justified  for  each 
fiscal  year. 
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In  order  to  ensure  that  ICS  is  properly  implemented 
and  that  adequate  funding  will  be  available  to  support  ICS 
requirements  when  needed,  ICS  planning  must  be  fully 
integrated  with  the  weapon  system  acquisition  life  cycle 
(see  Figures  1.1  and  1.2). 


Weapon  System  Acquisition  Life  Cycle 


Each  Air  Force  major  command  is  responsible  for  a 
specific  Defense  Mission  Area  and  must  continuously  analyze 
its  mission  capabilities  with  respect  to  the  evolving 
threat,  and  identify  operational  needs.  When  such  compre¬ 
hensive  mission  analysis  leads  to  the  identification  of 
an  operational  need  which  cannot  be  met  within  existing 
capabilities  and  may  require  new  weapon  system  or  equipment 
development,  a  Statement  of  Operational  Need  (SON)  may  be 
prepared.  Prior  to  publishing  a  SON,  the  major  command 
must  review  information  on  the  technology  base,  integrated 
logistics  support,  life  cycle  cost,  safety  and  training 
considerations  provided  by  the  originator.  At  this  point, 
the  evaluation  of  logistics  support  alternatives  must  be 
included  to  the  extent  that  information  is  available. 
Planning  for  possible  ICS  should  begin  at  this  point,  and 
remain  a  consideration  throughout  the  remainder  of  the 
weapon  system  acquisition  process. 
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Figure  1.2.  ICS  Tasks  Integrated  with  Major  System  Acquisition  Life  Cycle  (11) 
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Figure  1.2  continued 


The  SON  is  submitted  to  Headquarters  United  States 
Air  Force  (HQ  USAF)  for  review,  along  with  a  plan  for  the 
conceptual  phase  of  the  program.  A  justification  of  Major 
System  New  Start  (JMSNS)  is  required  for  all  acquisition 
programs  for  which  the  estimated  costs  exceed  $200M  ($FY80) 
for  research,  development,  test  and  evaluation,  and/or  $1B 
($FY80)  for  procurement.  Such  a  program  is  designated  a 
"major  system  acquisition  program"  and  requires  Secretary 
of  Defense  (SECDEF)  approval. 

The  JMSNS  is  submitted  in  the  Air  Force  Program 
Objective  Memorandum  (POM)  (Air  Force  input  to  the 
congressional  budget  cycle).  If  it  is  included  in  the 
final  DOD  budget,  the  Air  Force  is  authorized  to  begin 
concept  exploration.  (The  Secretary  of  the  Air  Force  may 
have  discretionary  funds  available  to  start  the  concept 
phase  prior  to  congressional  approval.)  If  the  program 
is  approved,  HQ  USAF  provides  formal  direction  to  the 
implementing  and  participating  commands  through  the 
issuance  of  a  Program  Management  Directive  (PMD ) .  The  PMD 
provides  the  formal  program  direction  throughout  the 
acquisition  process. 

Conceptual  Phase 

Once  the  PMD  is  issued,  the  Conceptual  Phase  begins. 


During  this  phase,  the  Air  Force  identifies  and  explores 
alternative  solutions  to  the  requirements  addressed  in  the 


SON.  A  Request  for  Proposal  (RFP )  is  issued,  soliciting 
contractor  solutions  in  the  form  of  technical  proposals. 

The  RFP  must  address  all  known  requirements,  including 
logistics  support  requirements.  The  Deputy  Program 
Manager  for  Logistics  (DPML)  must  prepare  the  Section  Nine 
Logistics  portion  of  the  Program  Management  Plan  (PMP). 

This  is  the  initial  version  of  the  Integrated  Logistics 
Support  Plan  (ILSP)  which  will  become  a  separate  document 
in  the  Demonstration/Validation  Phase.  The  PMP  Section 
Nine/ILSP  must  include  a  statement  of  ICS  objectives. 

During  this  phase,  the  DPML  must  also  identify  ICS  budget 
requirements  for  later  phases. 

Selection  of  the  alternative ( s )  for  demonstration  is 
documented  in  a  System  Concept  Paper  (SCP).  The  SCP  must 
contain  a  logistics  annex,  which  includes  ICS  requirements. 
The  SCP  is  reviewed  by  the  Air  Force  Systems  Acquisition 
Review  Council  ( AFSARC )  and  the  Defense  Systems  Acquisition 
Review  Council  (DSARC ) .  The  Secretary  of  the  Air  Force 
requests  SECDEF  approval  for  Milestone  I  (Requirement 
Validation  Decision)  to  enter  into  the  Demonstration  and 
Validation  Phase. 

The  Milestone  I  decision  is  a  SECDEF  validation  of 
the  requirement  for  the  major  system.  The  AFSARC  and 
DSARC  are  advisory  bodies  which  provide  information  and 
recommendations  to  the  Secretary  of  the  Air  Force  and  the 
SECDEF,  respectively,  in  support  of  their  decisions  at 


Milestones  I,  II,  and  III.  The  SECDEF  prepares  a  Secretary 
of  Defense  Decision  Memorandum  ( SDDM)  which  reaffirms  the 


mission  need  and  approves  one  or  more  selected  alternatives 
for  competitive  demonstration  and  validation. 

The  SDDM  documents  the  SECDEF' s  milestone  decision 
'  including  approval  of  goals  and  thresholds  for 

cost,  schedule,  performance,  and  supportability 
against  which  the  program  must  be  managed  and  will 
be  evaluated  [12:21]. 

Demonstration  and  Validation  Phase 

HQ  USAF  revises  the  PMD  to  provide  direction  to  the 
implementing  command  (usually  AFSC)  and  participating 
commands  for  the  Demonstration  and  Validation  (Demo/Val) 
Phase.  The  preferred  approach  to  Demo/Val  is  to  "select  at 
least  two  contractors  to  build  prototypes  which  are  to  be 
evaluated  during  the  latter  days  of  this  phase  [12:22]." 

During  the  Demo/Val  Phase,  the  DPML  must  prepare  the 
initial  ILSP,  including  ICS  requirements,  the  logistics 
Statement  of  Work  (SOW)  including  ICS  requirements,  and 
|  the  logistics  budget,  including  ICS  requirements.  The 

contractor ( s )  must  prepare  an  initial  Integrated  Support 
Plan  (ISP)  including  an  ICS  annex  or  ICS  Plan  (ICSP), 
perform  an  initial  LSA  to  identify  ICS  candidate  items 
and  organic  logistics  support  requirements,  and  submit 
a  technical  and  cost  proposal.  It  is  essential  that  Air 
Force  evaluation  of  such  proposals  be  thorough  during 
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this  phase,  since  competition  will  keep  proposed  costs 
low.  This  is  the  time  to  lock  in  the  ICS  requirements 
identification  and  planning  effort.  Supportability  and 
cost-effectiveness  must  be  given  as  much  consideration 
as  operational  performance  when  evaluating  proposals. 

A  Decision  Coordinating  Paper/Integrated  Program 
Summary  (DCP/IPS)  is  prepared,  summarizing  the  Air  Force 
acquisition  plan  for  the  system  life  cycle  and  providing 
a  management  overview.  After  an  AFSARC/DSARC  review,  the 
SECDEF  prepares  a  Secretary  of  Defense  Decision  Memorandum 
(SDDM) .  If  the  SECDEF  approves  the  program,  it  enters  Full 
Scale  Development  (FSD ) .  This  is  the  Milestone  II,  Program 
Go-Ahead  Decision. 


Full  Scale  Development  Phase 

During  the  FSD  Phase,  the  system  is  "designed, 
developed,  fabricated  and  tested  [12:25]."  Historically, 
support  equipment,  trainers,  technical  manuals,  and  other 
logistics  resources  have  been  developed  in  this  phase, 
also.  The  frequent  incidence  of  concurrent  FSD  and 
production  programs  in  recent  years  has  lead  to  support- 
ability  problems  resulting  in  the  use  of  ICS  (often  without 
adequate  planning).  During  the  FSD  Phase,  long-lead 
procurement  of  parts  and  materials  for  production  is  also 


authorized,  as  well  as  procurement  of  items  necessary  to 
support  test  programs. 

During  FSD,  Developmental  Test  and  Evaluation  ( DT&E ) 
and  Initial  Operational  Test  and  Evaluation  (IOT&E) 
efforts,  which  may  have  begun  in  earlier  phases,  escalate, 
culminating  in  qualification  tests  at  the  Milestone  III 
Decision  timeframe  (see  Figure  1.3).  "By  the  end  of'  this 
phase  detailed  design  specifications  will  be  finalized 
and  engineering  drawings  prepared  which  become  the  basis 
[12:25]"  for  the  production  configuration.  The  system 
design  is  stabilized  at  the  Critical  Design  Review  (CDR ) . 

In  most  system  programs,  design  changes  after  CDR  result  in 
the  generation  of  engineering  change  proposals  (ECP)  which 
bear  a  cost  to  the  government  for  implementation.  An  ECP 
is  "a  formal  proposal  to  alter  the  physical  or  functional 
characteristic  of  a  system  or  item  after  the  baseline 
configuration  has  been  established  [25:42]." 

During  the  FSD  Phase,  the  DP ML  must  update  the 
logistics  and  ICS  budgets,  the  ICS  SOW  and  the  ICS  annex  to 
the  ILSP.  The  contractor  must  update  the  ISP,  ICSP,  and 
LSA  Plan  (LSAP).  He  must  perform  a  detail  LSA,  refining 
ICS  and  organic  support  requirements.  After  CDR  for  the 
PME ,  the  support  equipment  designs  can  be  stabilized,  and 
also  undergo  a  CDR.  It  is  possible  that  a  number  of  design 
changes  or  ECPs  will  be  generated  during  CDR,  and  these 
changes  may  impact  support  equipment  design  and  technical 
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Milestone  Milestone  Milestone 


Figure  1.3  Test  and  Evaluation  Cycle 


manual  content.  The  DPML  must  be  certain  to  obtain 
detailed  LSA  updates  and  refine  the  ICS  requirements  before 
entering  the  production  phase. 

Production/Deployment  Phase 

"The  Secretary  of  the  Air  Force  has  the  authority 
to  make  the  Production/Deployment  (Milestone  III)  decision 
provided  there  is  no  major  change  to  the  program  approved 
at  Milestone  II  [12:28]."  If  approved,  the  decision 
is  implemented  by  a  revision  of  the  PDM  (12:28).  During 
the  Production  Phase,  the  weapon  system  and  all  related 
logistics  resources  are  produced  for  operational  use. 

During  this  phase,  any  logistics  decisions  deferred  from 
FSD  must  be  finalized.  The  ILSP  and  ICS  annex  are  updated 
one  last  time.  With  system  deployment,  ICS  and  organic 
support  are  implemented.  During  initial  deployment,  the 
DPML  and  contractor  must  work  together  to  transition  from 
ICS  to  organic  support  as  planned  in  the  PMD  and  ILSP. 
Frequently,  development  of  an  organic  depot  support 
capability  lags  behind  PME  development.  Depot  repair  is 
at  a  much  more  detailed  level,  and  development  of  depot 
support  equipment  must  be  deferred  until  PME  is  fully 
design  stable.  In  some  concurrent  programs  this  may  occur 
after  the  system  is  fielded.  Depot  ICS  is  usually  the  last 
portion  of  ICS  to  be  transitioned  to  the  Air  Force.  The 


actual  length  of  an  ICS  program  varies  on  a  case-by-case 
basis  and  is  determined  by  individual  program  requirements. 


Logistic  Support  Analysis  (LSA)  Process 

ICS  requirements  are  derived  primarily  from  LSA.  The 
requirements  are  expressed  as  candidate  items,  required 
tasks,  and  time  frames  and  schedules.  The  ICS  plan  and  the 
LSA  Plan  address  the  relationship  between  the  LSA  and  the 
ICS  requirements  determination  process.  A  detailed 
description  of  the  LSA  process  is  outside  the  scope  of  this 
thesis  and  can  be  found  in  MIL-STD  1388,  "Logistic  Support 
Analysis,  Data  Element  Definitions"  (23).  The  LSA  is  an 
iterative  process,  repeated  in  increasing  depth  as  the 
equipment  design  stabilizes  over  the  acquisition  life 
cycle.  ICS  requirements  are  developed  primarily  during  the 
FSD  Phase,  but  LSA  results  from  other  phases  also  impact 
ICS  decisions  [11s 2-14],  The  LSA  is  not  used  only  to 
identify  ICS  requirements.  It  is  used  to  identify  a  full 
range  of  maintenance  and  support  tasks,  resources  (i.e., 
piece  parts,  tools  and  equipment)  and  schedules. 

The  LSA  Record  (LSAR)  is  the  central  repository 
of  all  data  generated  in  the  LSA  process.  It  is  the 
"single  authoritative  source  of  integrated  technical  data 
pertaining  to  operational  logistics  support  [11:2-27]." 

ICS  status  reports  are  generated  from  the  LSAR,  in 


increasing  detail  as  design  data  stabilizes.  LSA  data 
identifies  the  support  resources  required  for  ICS  and 
organic  logistic  support,  including  support  and  test 
equipment,  spares,  repair  parts,  tools,  and  maintenance 
personnel  skill  levels.  This  information  is  generated 
as  it  becomes  available,  and  lead  times  to  procure  the 
required  resources  become  critical  as  the  IOC  date 
approaches.  Thus,  LSA  must  be  included  early  in  the  weapon 
system  acquisition  life  cycle,  and  updated  frequently 
as  the  PME  design  stabilizes  in  order  to  ensure  a  full 
logistics  support  capability  exists  when  the  system  is 
fielded  at  IOC. 

Conclusion 

Proper  planning  for  ICS  depends  on  full  integration 
of  that  planning  along  with  planning  for  the  organic 
logistics  support  capability,  into  the  weapon  system 
acquisition  life  cycle.  One  program  which  is  currently 
demonstrating  how  that  integration  is  possible  is  the  Air 
Force  HH-60D  Nighthawk  Helicopter  (7).  Mr.  C.  Wayne  Cerny 
of  IBM  Corporation  and  Mr.  David  Cuppett  of  the  HH-60D  DPML 
office,  have  co-authored  a  paper  detailing  their  program's 
innovative  approach  to  planning  and  managing  for  logistics 
supportability. 


Cerny  and  Cuppett  believe  that,  all  too  frequently, 
logistics  managers  "plunge  headlong  into  the  details  of 
logistics  documentation  well  before  the  engineering  design 
(of  the  prime  mission  equipment  [PME] )  is  mature  [7:2]," 
resulting  in  what  they  call  "throw-away"  logistics,  or 
"reams  of  data  and  early  hardware  procurements  which,  due 
to  the  normal  evolutionary  nature  of  system  design,  simply 
no  longer  reflected  the  latest  configuration  [7:3]."  They 
believe  that  such  "throw  away"  logistics  result  from 
improper  use  of  LSA,  and  too  early  development  of  logistics 
support  resources. 

Key  to  their  philosophy  is  the  development  of  an 
integrated  diagnostics  capability,  which  will  be  reflected 
in  the  LSA  data.  Cerny  and  Cuppett  also  believe  that 
timing  of  the  LSAR  is  critical,  so  they  planned  the  HH-60D 
LSAR  to  "complement  the  evolving  design  [7:6]."  During 
FSD,  only  organizational  support  equipment  (OSE)  is  being 
reviewed.  The  OSE  will  be  developed  by  the  contractor  as  a 
capability  and  demonstrated  during  IOT&E.  Intermediate  SE 
(ISE)  design  will  take  place  during  the  first  production 
lot  buy.  Depot  SE  (DSE)  design  is  planned  for  the  second 
production  lot  buy.  Firm-priced  options  for  ICS  were 
obtained  during  the  competitive  source  selection,  and  will 
be  exercised  as  needed  until  organic  logistics  support 
capability  exists.  Development  of  other  logistics  support 


resources  such  as  technical  data,  spares,  training,  and 
repair  parts  is  also  dependent  on  the  LSAR. 

To  summarize  their  approach,  Cerny  and  Cuppett  state, 


Proper  timing  of  the  various  support  elements  and 
the  associated  provisions  for  interim  contractor 
support  during  the  development  of  organic  capa¬ 
bility  simply  follows  the  logical  progression  as 
dictated  by  the  evolving  hardware  maturity.  If  we 
can  avoid  the  need  to  redo  a  significant  portion 
of  the  supportability  program,  the  actual  support 
will  be  available  when  required  at  an  overall  lower 
cost.  ...  To  enhance  the  probability  of  achieving 
the  supportability  objective,  we,  as  logistics 
professionals,  must  carefully  avoid  the  pitfall  of 
doing  too  much  too  early  [7:8-9]. 


The  complexity  of  modern  weapon  systems  and  the  lead 
times  required  in  the  budget  cycle  and  the  acquisition 
process  make  early  planning  for  ICS  critical.  The  cost- 
effective  development  of  logistics  support  resources 
appears  to  be  very  dependent  on  the  prior  achievement  of 
PME  design  stability.  Thus,  design  stability  must  be 
defined. 


Problem  Statement 


Currently,  there  is  no  precise  method  for  determining 
the  status  of  equipment  design  stability  as  it  is  used  in 
APR  800-21  or  for  any  other  application  to  the  development 
of  an  organic  logistics  support  capability.  Consequently, 
it  is  very  difficult  to  comply  with  the  requirement  to 
conduct  risk  analyses  and  identify  potentially  design 


unstable  components.  To  date,  very  little  effort  has  been 
expended  on  the  establishment  of  criteria  or  standards  to 
be  used  as  a  baseline  for  the  determination  of  weapon 
system  and  equipment  design  stability.  The  studies  which 
have  been  conducted  apply  only  to  specific  weapon  systems 
and  cannot  be  generalized  to  other  systems  or  equipment. 
The  most  frequently  used  indicator  of  design  stability  is 
the  rate  of  generation  of  engineering  change  proposals 
(ECPs) . 


The  point  at  which  the  rate  of  ECPs  generated 
is  reduced  to  near  zero  is  generally  assumed  to 
coincide  with  the  achievement  of  a  stable  design 
for  purposes  of  developing  logistics  support 
resources  [18]. 


There  is  a  need  for  an  analysis  of  the  feasibility  of 
using  design  stability  as  a  decision  parameter  for  making 
logistics  supportability  decisions. 


Purpose  of  the  Research 


The  purpose  of  this  research  was  to  investigate 
the  criteria  and  analysis  techniques  which  are  currently 
being  used  to  evaluate  design  stability  as  it  relates 
to  decisions  concerning  the  establishment  of  an  organic 
logistics  support  capability  versus  the  use  of  ICS. 
Specifically,  this  research  evaluated  the  use  of  the 
rate  of  generation  of  ECPs  as  an  indicator  of  weapon 


1.  Determine  if  there  is  a  trend  in  the  rate  of 
generation  of  ECPs  which  can  be  measured  over  time  and 
against  program  milestones,  indicating  when  design 
stability  has  been  achieved.  (Trend  as  used  here  refers  to 
dependencies  in  the  data  over  time  due  to  autocorrelations 
and  periodicity. ) 

2.  If  there  is  a  trend,  investigate  the  application 
of  the  trend  to  all  like  systems  as  a  predictor  of  the 
point  at  which  design  stability  will  be  achieved. 

3.  If  there  is  a  trend,  investigate  its  application 
to  those  ECPs  which  have  a  direct  impact  on  logistics 
support  resource  development. 

4.  Investigate  the  adequacy  of  existing  methods  of 
determining  whether  or  not  a  design  is  stable  for  purposes 
of  developing  organic  logistics  support  capability. 

Scope  and  Limitations 

This  research  effort  addresses  only  the  F-16,  F-15, 
and  EF-lllA  aircraft.  Only  Class  I  ECPs  were  used  in  the 
ECP  trend  analysis,  which  was  limited  to  the  F-16  and  F-15 


fire  control  and  flight  control  systems  and  the  EF-111A 
avionics.  Interviews  were  conducted  to  obtain  current 
information  on  the  subject  of  design  stability  from  AFALD 
DPMLs  involved  in  the  acquisition  of  ICS  and  organic 
logistics  support  resources,  and  from  aerospace  industry 
personnel  who  provide  such  resources.  As  such,  the 
responses  are  limited  in  scope  to  that  population,  and 
cannot  be  generalized  to  other  situations  or  populations. 
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CHAPTER  II 


BACKGROUND  AND  LITERATURE  REVIEW 


Recent  government  and  industry  studies  have  deter¬ 
mined  that  design  stability  from  a  logistics  supportability 
standpoint  is  relatively  undefined,  and  that  cost  and  risk 
analyses  as  required  by  AFR  800-21  are  not  well-documented 
by  program  offices.  The  Air  Force  Audit  Agency  conducted  a 
review  of  programs  relying  on  ICS.  The  results  of  this 
review  were  published  in  a  final  report  dated  3  September, 
1982.  The  report  cites  a  need  for  improved  "documentation 
of  planning  and  analysis  supporting  ICS  decisions  [20:i],n 
and  states  that. 


Acquisition  program  managers  did  not  have  documen¬ 
tation  to  adequately  support  ICS  decisions  for  9  of 
14  programs  reviewed.  Consequently,  management 
could  not  demonstrate  that  early  planning  or  cost 
and  risk  analysis  were  accomplished  for  these  pro¬ 
grams  which  accounted  for  $151  million  of  projected 
ICS  requirements  for  fiscal  years  1982-1984  [20:8], 


Since  such  analyses  were  not  completed,  logistics  managers 
could  not  substantiate  that  ICS  was  the  most  cost  effective 
logistics  support  alternative.  The  audit  team  believed  the 
absence  of  specific  directive  guidance  to  be  the  cause  of 
this  lack  of  documentation. 


25 


Logistics  managers  agreed  with  this  evaluation, 
but  believed  that  the  establishment  of  specific,  minimum 
documentation  requirements  would  also  lead  to  the  unin¬ 
tentional  establishment  of  maximum  requirements.  The 
logistics  managers  maintained  that  "each  program  has  unique 
considerations  that  must  be  analyzed  on  a  case-by-case 
basis,”  and  that  "an  open-ended  requirement  will  result 
in  more  specific  and  better  quantified  studies  [20:9-10]." 
The  audit  agency's  final  comment  was  that  "analysis  on  a 
case-by-case  basis  will  be  effective  if  HQ  AFSC  establishes 
appropriate  controls  to  ensure  the  analysis  is  documented 
[20:10]." 

In  order  to  perform  the  cost  and  risk  analyses 
required  by  AFR  800-21/  logistics  managers  must  be  able  to 
identify  certain  parameters.  They  must  know  what  the  final 
prime  mission  equipment  design  will  look  like  before  they 
can  begin  to  develop  support  equipment  to  test,  maintain, 
and  repair  it.  In  his  study  of  "Baseline  Indicators  of 
Production  Readiness,"  (3),  published  in  November  1980,  Mr. 
John  Bemis  identified  many  hardware  and  software  indicators 
of  production  readiness.  One  of  those  indicators  is  the 
rate  of  generation  of  ECPs.  He  states  that,  "Examination 
of  the  profiles  of  engineering  change  traffic  can  be 
revealing  in  terms  of  the  design  maturity  of  a  system  [3]." 

The  curves  depicting  ECP  rates  plotted  over  time  should 
follow  a  downward  trend.  "Sustained  levels  of  high  change 


rate  indicate  a  risk  to  cost,  schedule,  and/or  performance 
131."  Since  development  of  logistics  elements  follows 
development  of  prime  mission  equipment  (PME) ,  it  is  logical 
to  assume  that,  while  the  PME  is  not  design  stable,  there 
would  be  risk  (both  cost  and  schedule)  associated  with  the 
development  of  logistics  elements  also.  Bemis  found  that 
empirical  data  compiled  during  his  study  indicated  that 

the  shape  of  the  engineering  change  traffic  pro¬ 
file  was  of  a  similar  shape  for  different  kinds 
of  systems  including  aircraft,  electronic  systems, 
tracked  vehicles,  and  gun  systems.  The  profile 
is  sufficiently  defined  such  that  anomalies  can 
be  identified  and  investigated  [3]. 

Thus,  it  would  seem  that  this  method  could  be  used  to 
determine  design  stability  for  a  new  system,  by  comparison 
to  existing  systems  of  a  similar  type. 

The  ARINC  Research  Corporation  conducted  a  study  of 
the  impact  of  ECPs  on  logistics  support  for  the  Navy  P-3 
aircraft.  Published  in  December  1972,  the  study  assessed 
the  impact  of  both  avionics  and  airframe  ECPs  on  spares 
support,  pub1 ications,  training,  and/or  ground  support 
equipment.  The  following  data  was  included  in  the  study 
(2:3)  : 

1.  ECP  number 

2.  Lockheed  submission  date  (month,  year) 

3.  Affected  aircraft  (P-3A/B/C) 

4.  Affected  system  description 
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5.  Change  control  board  number 

6.  Change  control  board  date  (month,  year) 

7.  Change  control  board  action  (approved, 
disapproved ) 

8.  Contract  amendment  date  (month,  year) 

9.  Type  of  engineering  change  (airframe, 
avionics,  power  plant,  etc.) 

10.  Technical  directive  number 

11.  Technical  directive  date  (month,  year) 

12.  Incorporation  status  (forward  fit  only, 
or  retrofit  and  forward  fit) 

13.  Estimated  cost 

14.  Description  of  ECP,  including  consideration 
of  inpact  on  a)  spares;  b)  publications; 

c)  training;  and  d)  ground  support  equipment 

Inpact  on  spares  support  was  further  categorized  as 
follows  (2:4): 

Minimum  Impact.  The  ECP  being  reviewed  results  in  no 
modification  to  existing  sparing.  It  has  no  effect  on  the 
quantity  of  spares  procured  previously  or  on  order. 

Average  Impact.  The  ECP  being  reviewed  results  in  a 
minor  modification  to  existing  spares-procurement  policy. 

It  does  not  require  scrapping  any  portions  of  existing 
inventory. 


Maximum  Impact.  The  ECP  being  reviewed  results  in  a 
major  modification  to  existing  sparing  policy.  This 
modification  can  take  one  of  the  following  forms: 

1.  Requirement  to  scrap  portions  of  existing 
inventory 

2.  A  change  in  spares  procurement,  necessitating 
additional  spares  purchases 

3.  A  change  in  existing  maintenance  concept 

Impact  on  training,  publications,  and  ground  support 
equipment:  in  these  three  areas,  the  prime  consideration 
was  an  evaluation  of  whether  the  ECP  did  or  did  not  have  an 
impact  on  the  specified  area. 

After  a  preliminary  analysis  of  all  available  data, 
it  was  determined  that  the  Lockheed  submission  date  was  the 
only  date  "with  sufficient  coverage  available  [2:5] . n 
Therefore,  that  date  was  used  as  the  common  base  date  for 
all  frequency  analyses.  The  data  analysis  was  then 
performed  using  all  possible  combinations  of  aircraft 
types,  and  the  various  types  of  ECPs  (avionics,  airframe, 
power  plant,  etc.). 

The  initial  output  was  "a  set  of  frequency 
distributions,  one  set  per  type  of  change,"  showing  the 
following  (2:6): 

Cumulative  number  of  relevant  ECPs 

Percentage  of  ECPs  having  maximum  spares  impact 


Percentage  of  ECPs  having  average  spares  impact 
Percentage  of  ECPs  having  minimum  spares  impact 
Percentage  of  ECPs  having  publications  impact 
Percentage  of  ECPs  having  training  impact 
Percentage  of  ECPs  having  ground  support 
equipment  impact 

Evaluation  of  the  results  led  to  the  division  of  the 
data  into  two  categories,  P-3A/B  and  P-3C,  since  the  P-3A 
and  P-3B  are  very  similar  and  very  different  from  the  P-3C 
(The  difference  is  primarily  in  the  avionics  suite.  )  The 
P-3A/B  can  be  categorized  as  a  mature  system,  and  the  P-3C 
cannot  (2:6). 

The  data  analysis  showed  that  as  the  aircraft  system 
matured,  for  both  avionics  and  airframe  ECPs, 


There  was  an  increase  in  the  relative  number  of 
ECPs  having  maximum  impact  on  spares.  .  .  .  How¬ 
ever,  the  absolute  percentage  having  maximum  impact 
never  exceeded  ten  percent  for  the  P-3A/B,  and  the 
remainder  of  the  submitted  airframe  and  avionics 
ECPs  were  evenly  split  between  average  and  minimum 
impact  [2:6]. 


The  P-3C  avionics  ECPs  "maximum  impact  reached  twenty 
percent,  possibly  reflecting  problems  with  the  more 
advanced  avionic  subsystems  [2:10]."  These  figures  can  be 
used  to  make  estimates  of  impacts  of  future  ECPs  on 
logistics  elements.  They  a: so  demonstrate  that  "there  is 
a  definite  leveling  of  ECP  activity  as  the  aircraft  system 
matures  [2:10]." 


»: 


Conclusions  of  the  ARINC  study  are  as  follows: 


1.  On  the  basis  of  the  P-3A/B  ECP  data,  it 
appears  that  the  number  of  ECPs  submitted  by  the 
prime  contractor  approaches  zero  as  the  end  of 
production  is  reached. 

2.  The  number  of  ECPs  being  submitted  each 
year  on  the  P-3C  is  still  increasing,  particularly 
for  the  avionic  subsystems,  possibly  reflecting 
their  complexity  in  comparison  with  the  P-3A/B 
avionic  subsystems. 

3.  There  appears  to  be  no  central  location 
where  complete  ECP  data  are  available. 

4.  Only  a  small  percentage  (ten  percent) 
of  ECPs  submitted  had  maximum  impact  on  spares 
support . 

5.  A  high  percentage  (ninety  percent)  of  ECPs 
submitted  affected  publications  [2:13]. 


Recommendations  of  the  ARINC  study  are  as  follows: 


1.  A  study  similar  to  this  one  should  be  con¬ 
ducted  on  an  aircraft  that  has  recently  completed 
the  production  phase  of  its  life  cycle.  This 
additional  study  should  be  used  to  verify  the 
results  of  the  P-3  ECP  Trend  Analysis  and  to  per¬ 
mit  these  results  to  be  extended  to  other  aircraft 
types . 

2.  A  central  repository  of  ECP  data  should  be 
established  and  maintained  within  NAVAIR.  As  a 
minimum,  this  system  should  be  established  for  the 
P-3C  to  continue  the  monitoring  of  ECPs  initiated 
in  this  study  [2:13]. 


Another  study,  conducted  by  the  Aerospace  Industries 
Association  of  America,  Inc.  (AIA),  in  October,  1979, 
revealed  that  while  much  effort  has  been  expended  on  the 
topic  of  design  stability  from  a  logistics  supportability 


standpoint,  most  of  it  focused  on  a  specific  weapon  system 
or  problem.  The  study  consisted  of  two  parts.  A  survey 
questionnaire  was  distributed  to  members  of  the  AIA,  and 
the  results  were  analyzed  for  trends.  Also,  a  quantitative 
analysis  of  design  change  activity  as  related  to  program 
milestones  was  performed  on  data  from  the  Navy  S-3A  weapon 
systems  program.  This  analysis  was  conducted  to  determine 
the  optimum  period  to  accomplish  support  responsibility 
transfer  of  new  aircraft  programs  from  the  contractor  to 
the  Navy. 

Results  of  the  questionnaire  indicated  that  many  of 
the  industries  surveyed  had  conducted  in-house  studies  to 
identify  a  method  for  assessing  design  stability.  The 
indicator  most  often  used  by  those  industries  to  assess 
design  stability  from  a  logistics  supportability  standpoint 
is  the  point  at  which  the  number  of  ECPs  has  been  reduced 
to  near  zero  (18).  The  data  suggests  the  number  of  ECPs 
generated  per  month  builds  to  a  peak  during  prototype 
development,  with  a  smaller  peak  during  prototype  testing 
(18).  This  finding  concurs  with  that  of  the  Bemis  study, 
which  was  noted  earlier  in  this  discussion  (3).  Another 
result  of  the  AIA  study  is  the  consensus  that  delays  in 
achieving  design  stability  can  have  a  significant  impact  on 
the  development  of  logistics  support  elements,  especially 
in  the  cost  area  (18). 


The  factors  which  were  considered  to  have  the  most 
impact  on  a  weapon  system/equipment  achieving  logistics 
supportability  as  related  to  design  stability  are  listed 
below,  as  prioritized  by  the  survey  respondents  (1): 

Number  and  duration  of  design  change  activity 
Funding 

Complexity  of  equipment 
Support  equipment  development 

Length  of  full-scale  engineering  development  phase 

Validation  of  logistics  requirements 

Parts  replacement  rates 

Availability  of  resources 

First  article  delivery  requirements 

Future  technology  vs.  state  of  the  art 

IOC 

Test  history 
Life  cycle  costs 
Length  of  concept  phase 
Type  of  contract 

Four  of  the  first  five  most  significant  factors 
relate  directly  to  design  stability  which  is 
identified  as  having  the  greatest  impact  on 
supportability  during  the  critical  introductory 
phase  of  a  new  weapon  system  [1]. 

Respondents  cited  many  situations  which  forced  the 
acceptance  of  an  unstable  design  and  resulted  in  the  use  of 
ICS,  including  the  following:  1)  urgent  need  by  customer 
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(forced  production  of  an  unstable  design);  2)  degree  to 
which  the  state  of  the  art  is  being  pushed  by  the  design; 
and  3)  funds  availability  (18).  These  situations  are  not 
truly  conditions  which  may  be  used  to  justify  a  requirement 
for  ICS  according  to  AFR  800-21.  All  of  them  appear  to  be 
situations  which  would  warrant  the  type  of  cost  and  risk 
analyses  required  by  AFR  800-21.  They  also  appear  to  be 
situations  which  require  individual  (case-by-case)  analyses 
and  solutions. 

The  AIA  survey  did  not  result  in  the  establishment 
of  any  criteria  for  the  determination  of  design  stability. 
It  did  identify  an  "increasing  acceptance  on  the  part  of 
DOD  acquisition  agencies  to  contract  for  longer  periods 
of  ICS"  which  "allows  for  a  significant  increase  in  the 
quality  and  a  decrease  in  the  cost  of  the  delivered 
logistics  elements  [18]."  Finally, 


Effective  planning  and  implementation  of  ICS 
is  recommended  as  the  best  approach  to  provide 
logistics  support  services  during  periods  of  design 
instability  and  the  time  period  needed  to  establish 
organic  support  capability  [18], 


Transfer  of  support  responsibilities  from  the 
contractor  to  the  Navy  depends  upon  two  basic 
factors:  (1)  that  point  in  time  when  the  rate  of 
engineering  change  approaches  a  steady  rate  and 
the  design  becomes  stable,  and  (2)  that  point  in 
time  when  the  Navy  has  developed  its  resources 
(skills  and  facilities  to  support  operational 
units).  Attainment  of  these  factors  should 
coincide  with  a  major  milestone  of  the  program 
identified  as  the  Naval  Support  Date  ( NSD)  [1], 


The  analysis  of  the  data  showed  that  "the  greatest  design 
change  occurred  ...  at  the  program  NSD  milestone  [1]." 

The  study  panel  concluded  that  "design  stability  of  a  new 
weapon  system  cannot  occur  until  the  rate  of  engineering 
change  (with  respect  to  time)  drops  sharply  and  approaches 
a  steady  state  value  [1]."  They  stated  that  the 

impact  of  design  change  activity  upon  systems/ 
equipment  can  be  significant  and  affects  logistics 
support  resources  such  as  maintenance  plans, 
spares,  technical  manuals,  ground  support  and 
test  equipment,  maintenance  training  and  trainers, 
and  facilities.  This  implies  that  the  maturity 
of  system  design  and  logistics  support  must  occur 
simultaneously  [1], 

They  recommended  realigning  major  program  milestones  to  a 
more  realistic  time  period  and  planning  for  a  longer  period 
of  interim  support  (1). 

In  November,  1981,  AFALD/XRG  conducted  a  study  of 
ICS  wartime  surge  capabilities  (16).  The  study  included 
a  survey  of  DPMLs  in  AFALD/SD  and  LW.  Responses  were 
received  from  forty  of  the  seventy-four  DPMLs.  One 
question  asked,  "Why  are  you  using  ICS?"  The  following 


responses  were  received: 

Design  Instability  1 

Concurrency  5 

Logistics  development  delayed  /  funds 
cut,  support  equipment  unavailable  9 

Depot  assignment  delayed  or  not  made  10 

Option  (use  if  needed)  6 


Not  needed 


3 


Requirements  unknown  at  present  1 

No  response  5 

These  responses  indicate  that  there  is  still  a  lot 
of  ICS  which  results  from  improper  or  lack  of  planning. 
Design  instability  was  cited  only  once  as  a  reason  for 
using  ICS,  and  is  the  only  reason  given  which  is  clearly 
defined  as  an  acceptable  reason  in  AFR  800-21.  Support 
equipment  unavailability  and  program  concurrency  could  be 
categorized  as  Situation  2,  where  lead  times  for  support 
resources  are  too  long  to  achieve  organic  logistics  support 
capability  by  IOC  (21).  The  six  respondents  who  stated 
that  ICS  is  an  option,  available  to  be  used  if  needed,  have 
obviously  planned  for  ICS. 

Conclusion 

The  results  of  the  studies  discussed  in  this 
literature  review  are  inconclusive  at  best.  Results  of 
both  the  ARINC  (2)  and  the  Bemis  (3)  studies  appear  to 
suggest  that  trend  analysis  of  ECPs  can  be  used  to  identify 
the  point  where  design  stability  is  achieved.  The  AIA 
study  (1)  supports  this  by  identifying  ECP  trend  analysis 
as  the  most  frequently  used  indicator  of  design  stability 
by  industry.  However,  Air  Force  logistics  managers  do  not 
appear  to  be  using  this  technique  in  the  ICS  planning 
process  for  weapon  system  acquisitions. 
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CHAPTER  III 


RESEARCH  METHODOLOGY 

This  research  effort  consisted  of  two  parts.  The 
first  part  was  a  trend  analysis  of  ECPs  and  their  impact 
on  logistics  support  elements.  The  second  part  was  a 
qualitative  study  of  the  adequacy  of  available  alternative 
methods  of  determining  the  point  at  which  design  stability 
is  achieved. 


ECP  Trend  Analysis 

The  ECP  trend  analysis  was  conducted  using  data 
collected  from  the  F-16,  P-15,  and  EF-111A  aircraft.  The 
data  collected  included: 

1.  ECP  number  and  title 

2.  ECP  description  of  change 

3.  ECP  impact  on  logistics  elements 

a.  spares 

b.  support  equipment 

c.  publications 

d.  training 

Only  Class  I  ECPs  were  considered  in  this  thesis 
because  of  their  possible  impact  on  logistics  elements. 


ECPs  are  classified  as  Class  I  in  MIL-STD-480  (25:2-3), 
when  one  or  more  of  the  following  factors  are  affected: 

1.  Functional  configuration  identification 

2.  Product  configuration  identification 

3.  Technical  requirements  (i.e.,  maintainability, 
reliability,  weight,  performance,  etc. )  which  are 
below  product  identification 

4.  Non-technical  contractual  provisions  (i.e.,  cost, 
schedules,  quantities,  etc.) 

5.  Other  factors  such  as  safety,  compatibility  with 
test  equipment,  interchangeability,  suitability, 
or  replaceability. 

Other  classes  of  ECPs  do  not  impact  these  factors, 
so  they  were  not  included.  The  fire  control  and  flight 
control  systems  on  the  F-15  and  F-16  were  selected  for 
analysis  because  they  are  very  similar,  and  provide  a  basis 
for  comparison.  The  EF-111A  data  were  collected  because, 
as  a  fighter  aircraft  avionics  modification,  compatibility 
was  possible.  Only  those  ECPs  which  directly  impacted 
specific  logistics  elements  were  included  in  the  data  base. 
(The  logistics  elements  included  were:  support  equipment, 
spares,  technical  data,  and  training.)  For  the  F-16  only 
those  ECPs  which  impacted  the  United  States  models  were 
included.  All  foreign  modifications  were  excluded.  Data 
sets  for  each  aircraft  type  consisted  of  the  quantity  of 
ECPs  approved  per  month,  arranged  in  chronological  order 


to  facilitate  identification  of  any  time  dependencies  in 
the  data. 


Research  Question  Number  One 

Research  question  number  one  stated,  "Determine  if 
there  is  a  trend  in  the  rate  of  generation  of  ECPs  which 
can  be  measured  over  time  and  against  program  milestones, 
indicating  when  design  stability  has  been  achieved."  To 
answer  this  question  the  data  were  evaluated  using  the 
Statistical  Package  for  the  Social  Sciences  (SPSS)  (10). 

The  Box-Jenkins  procedure  was  used  to  determine  if  the 
data  are  autocorrelated,  i.e.,  if  there  is  evidence  of 
time  dependencies.  Time-series  plots  of  the  number  of  ECPs 
per  month  were  generated  for  each  aircraft  type  and  are 
shown  in  Appendix  D.  With  these  plots  and  Box-Jenkins 
procedures,  one  can  deduce  any  trends  that  may  be  embedded 
in  the  data.  (Trends  as  used  here  refers  to  dependencies 
in  the  data  over  time  due  to  simple  auto-correlation  and 
periodicity.)  After  analyzing  the  results  of  this  step, 
the  existence  of  simple  autocorrelation  and  periodicity  was 
confirmed.  Thus,  an  observation  at  time  t  is  not 
independent  of  observations  prior  to  time  t.  This  finding 
precludes  the  use  of  linear  or  multiple  regression,  since 
the  residuals  cannot  be  assumed  to  be  independent.  Thus, 
the  data  were  evaluated  using  the  TIMES  time  series 


analysis  package  (26).  Based  on  analysis  of  the  auto¬ 
correlation  function  (ACF)  and  partial  autocorrelation 
function  (PACF)  of  each  time  series,  an  Autoregressive 
Integrated  Moving  Average  (ARIMA)  model  was  proposed  as 
a  model  of  the  time  series  for  each  weapon  system.  ARIMA 
models  of  order  (p,d,q)  were  used,  of  the  form: 

(1  -  (JjjB  -  <j>2B2  -  ...  -  <t>pBP)  (1  -  B)dZfc  * 

(i  -  e.B  -  e«B2  -...  -  e„Bq)  e. 

12  g  t 

Here,  d  denotes  the  degree  of  differencing  necessary  to 
convert  a  time  series  to  a  stationary  process  while  the  p 
and  q  are  the  order  of  the  autoregressive  and  moving 
average  components  (13:205-206).  The  operator  B  is  a 
backshift  operator  such  that  BnZt  =  Zfc_n.  The  least- 
squares  estimated  values  of  the  coefficients  0^  and  <{)j  were 
obtained  using  the  TIMES  computer  code 

This  analysis  adhered  to  the  principle  of  parsimony 
as  espoused  by  Box  and  Jenkins.  Models  were  selected  which 
adequately  described  the  time  series,  yet  contained  as  few 
parameters  as  possible  (13:192). 

Once  the  ARIMA  (p,d,q)  model  was  tentatively  identi¬ 
fied,  it  was  examined  for  adequacy,  or  goodness-of-f it. 

"If  the  fitted  model  is  adequate,  it  should  transform  the 
observations  to  a  white  noise  process  [13:211]."  This 
means  that  if  the  model  is  appropriate,  the  residual  sample 


autocorrelation  function  "should  not  differ  significantly 
from  zero  for  all  lags  greater  than  or  equal  to  one 
[13:211-212]."  A  "white  noise  process"  refers  to  a 
distribution  of  error  terms  which  has  a  mean  of  zero, 
constant  variance,  and  no  autocorrelation.  A  well-fitted 
ARIMA  model  reduces  the  error  component  to  a  white  noise 
(4).  A  lag  refers  to  the  constant  interval  between  the 
data  points  being  plotted,  i.e.,  the  autocorrelation  at  lag 
one  refers  to  the  autocorrelation  between  observations  one 
time  interval  apart.  In  order  to  test  for  white  noise,  the 
following  diagnostics  were  performed. 

Portmanteau  Lack  of  Fit  Test.  The  ARIMA  (p,d,q) 
models  were  tested  for  adequacy  using  the  Portmanteau  lack 
of  fit  test.  This  test  evaluates  the  smallness  of  an 
arbitrary  K  lags  of  a  sample  autocorrelation  function  (5). 
Specifically, 


Q 


2 

is  approximately  distributed  as  X  ,»  „  where  n  =  N  -  d 

\ «“P"4 * 

and  Tj  ■  autocorrelation  at  lag  j.  The  observed  value  of  Q 

2 

was  checked  against  the  X  value  with  the  appropriate 

degrees  of  freedom  at  a  -  .05.  If  Q  is  greater  than 
2 

X  (K_p_q)'  then  the  ACF  of  the  residuals  is  said  not  to  be 
white  noise  (4:290-291). 
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is  "specifically  designed  for  the  detection  of  periodic 


patterns  in  a  background  of  white  noise  [4:294]."  It 
was  used  to  identify  periodicities  in  the  residuals  noc 
accounted  for  by  the  model.  Periodograms  were  generated 
for  each  data  set  using  the  TIMES  package. 

Residuals  Histogram.  The  residuals  of  each  data 
set  were  plotted  on  a  histogram  to  test  for  normality. 
Normally  distributed  residuals  permit  probability  state¬ 
ments  regarding  the  probability  of  falsely  declaring  a 
time  series  to  be  autocorrelated  when  in  fact  it  is  white 
noise. 

Research  Question  Number  Two 

Research  question  number  two  stated:  "If  such  a  trend 
exists,  demonstrate  that  this  trend  can  be  applied  to  all 
like  systems,  so  that  it  can  be  used  as  a  predictor  of  the 
point  at  which  design  stability  will  be  achieved."  To 
answer  this  question,  the  time  series  plots  for  all  the 
aircraft  data  were  evaluated  to  determine  if  any  trends 
were  similar.  The  ARIMA  models  developed  for  each  type  of 
aircraft  were  analyzed  to  determine  if  they  could  be  used 
as  predictors  of  ECP  trends,  and  thus,  predict  achievement 
of  design  stability  for  new  systems. 


Research  Question  Number  Three 


Research  question  number  three  stated:  "If  there  is 
a  trend,  demonstrate  that  it  can  be  applied  to  those  ECPs 
which  have  a  direct  impact  on  logistics  support  element 
development."  To  answer  this  question,  ECPs  for  all 
aircraft  were  evaluated  to  determine  if  they  impacted 
logistics  support  elements.  Only  those  ECPs  which  did 
impact  logistics  support  elements  were  included  in  the 
analysis. 

Research  Question  Number  Four 

Research  question  number  four  stated:  "Investigate 
the  adequacy  of  existing  methods  of  determining  whether  or 
not  a  design  is  stable  for  purposes  of  developing  organic 
logistics  support  capability."  To  answer  this  question, 
results  of  the  ECP  trend  analysis  were  evaluated  to 
determine  the  adequacy  of  such  an  analysis  for  use  as  a 
measure  of  design  stability  for  logistics  supportability 
decisions. 


Qualitative  Analysis 

The  qualitative  analysis  was  conducted  using  data 
collected  from  a  series  of  interviews  with  logistics 
management  personnel  who  have  had  varying  amounts  of 


experience  managing  ICS  programs.  The  interview  sample 
consisted  of  both  military  and  civilian  personnel  from  the 
Air  Force  Acquisition  Logistics  Division,  Deputy  for 
Aeronautical  Programs  (AFALD/SD),  Wright-Pat ter son  AFB  OH, 
and  civilian  personnel  from  the  aerospace  industry.  The 
interview  sample  size  (eleven  people)  was  not  intended 
to  be  statistically  significant,  and  the  results  of  the 
interviews  are  useful  only  for  discussion  in  this  thesis 
and  as  a  basis  for  further  research.  The  sample  size  was 
deliberately  limited  because  of  time  constraints.  There¬ 
fore,  the  results  "cannot  be  generalized  to  any  population 
[19:81]"  other  than  the  sample  population. 

In  order  to  facilitate  the  interview  process,  a 
package  containing  an  interview  guide  and  a  worksheet 
(Appendix  B)  was  mailed  to  the  subjects  in  advance.  Each 
subject  was  then  contacted  by  telephone  to  arrange  an 
interview,  and  requested  to  prepare  the  worksheet  in 
advance.  Questions  on  the  worksheet  are  open-ended  and 
subjective.  The  interviews  followed  the  outline  of  the 
worksheet,  but  also  included  several  demographic  questions 
The  actual  interviews  followed  a  standard  format  and 
sequence  (Appendix  C),  thereby  assuring  that  each  question 
was  asked  the  same  way  in  each  interview.  Questions  built 
on  previous  responses,  which  added  to  the  reliability  of 
the  interview  technique  (8:215).  The  interview  guide  was 
validated  by  a  series  of  interviews  designed  to  evaluate 


the  respondent's  understanding  of  the  questions  and  their 
intent.  The  validation  interviews  were  conducted  using 
AFIT  instructors  as  subjects. 

The  questions  in  the  interview  guide  were  designed  to 
provide  information  which  will  aid  in  the  evaluation  of  the 
research  questions,  within  the  expressed  limitations  and 
scope,  and  in  the  identification  of  areas  requiring  further 
research.  The  responses  to  the  questions  in  the  interview 
guide  were  especially  pertinent  to  research  question  number 
four.  Responses  were  compared  and  categorized,  then 
conclusions  were  drawn  based  on  the  similarities  and 
differences  among  them.  Demographic  data  were  used  to 
ascertain  any  variance  among  responses  from  subjects  which 
could  be  attributed  to  amount  of  experience  in  managing 
logistics  support  and/or  ICS. 


CHAPTER  IV 


ANALYSIS  AND  RESULTS 

ECP  Trend  Analysis 

After  the  data  were  collected  for  the  P-15,  F-16,  and 
EF-111A  aircraft,  it  became  apparent  that  there  were  not 
sufficient  quantities  of  EF-111A  ECPs  which  had  an  impact 
on  the  logistics  elements.  The  EF-111A  data  was  therefore 
eliminated  from  this  thesis  effort.  (Box-Jenkins  time 
series  analysis  techniques  require  a  minimum  of  fifty  data 
points;  there  were  only  twenty-eight  data  points  available 
for  the  EF-lllA. )  Thus,  the  analysis  proceeded  using  only 
the  F-15  and  F-16  data. 

All  ECPs  generated  in  each  program  were  examined  and 
included  in  the  data  base  if  they  met  the  requirements 
stated  in  Chapter  III,  Research  Methodology.  The  F-15  data 
set  consisted  of  113  ECPs  and  the  F-16  data  set  consisted 
of  64  ECPs.  The  data  sets  consisted  of  the  quantity  of 
ECPs  approved  per  month,  arranged  in  chronological  order 
so  that  autocorrelation  could  be  identified.  The  research 
questions  stated  in  Chapter  III,  Research  Methodology, 
provided  the  framework  for  the  analysis. 


Research  Question  Number  One 


Research  question  number  one  stated  "Determine  if 
there  is  a  trend  in  the  rate  of  generation  of  ECPs  which 
can  be  measured  over  time  and  against  program  milestones, 
indicating  when  design  stability  has  been  achieved."  To 
answer  this  question,  the  data  were  evaluated  using  the 
SPSS  Box-Jenkins  procedure  to  determine  the  existence  of 
autocorrelation  in  the  data.  Both  data  sets  were  found 
to  contain  autocorrelation.  Next,  both  data  sets  were 
evaluated  using  the  TIMES  times  series  analysis  package. 

By  plotting  the  observed  series  deviation  from  the  mean, 
both  data  sets  were  determined  to  be  stationary,  so  no 
differencing  was  required  and  d  =  0  for  both  data  sets. 

The  ACF  and  PACF  plots  were  studied  for  patterns  of 
autocorrelation.  The  two  data  sets  exhibited  different 
patterns,  so  individual  models  were  developed  for  each 
aircraft.  Results  of  the  modeling  process  are  given  in 
Tables  4.1  and  4.2. 

The  F-15  data  exhibited  a  pattern  of  significant 
autocorrelation  at  lags  one,  three,  six  and  nine  (see 
Appendix  D).  Four  simple  models  were  tested:  AR(1),  AR(2) 
MA( 1 ) ,  and  MA(2).  Of  these  simple  models,  the  MA(1)  model 
best  fits  the  observed  data.  Using  the  Portmanteau  lack 
of  fit  test,  the  MA(1)  model  reduced  the  Q  statistic  the 
most  (see  Table  4.1).  However,  the  Q  statistic  value  of 


31.624  was  still  much  greater  than  X' 


10.85.  This 


model  eliminated  the  autocorrelation  at  lag  one,  but 
did  nothing  about  lags  three,  six  and  nine.  Two  new 
models  were  tested:  ARIMA  (0,0,1)*(0, 0,3)3,  and  ARIMA 
(0,0,1)*(3, 0,0)^.  These  two  models  both  reduced  Q,  but 
the  ARIMA  (0,0,1)*(0, 0,3)3  model  had  the  best  results 
( see  Table  4.1). 

The  final  model  for  the  F-15  data  is  an  ARIMA 
(0,0,1)*(0, 0,3)3  model  (4:305).  The  model  equation  (in 
Box-Jenkins  backshift  operator  notation)  is 


Zt  *  (1  +  . 15593B)  (1  +  .18405B3  +  .036158B6 


+  .33203B*)  e. 


Multiplying  out  the  right-hand  side  of  equation  [4.1] 


[4.1] 


yields  the  equation: 


Zfc  -  1  +  .15593Bet  +  .18405B3et  +  ,0294459B4et 
+  .036158B6et  +  .0056382B7et  +  .33203B9et 

+  .  0517692B10ej. 


Equation  [4.2]  can  be  expressed  in  more  conventional 
notation  by  removing  the  backshift  operators  to  yield 
the  final  equation: 


[4.2] 


Z  =  6«.  +  .155936.  .  +  .184056.  ,  +  .02944596.  . 
t  t  t-1  t-3  t-4 

[4.3] 

+  .0361586t_6  +  .00563826t_?  +  .332036t_9 
+  .05176926t_10 

The  F-16  data  base  contained  a  large  number  of  data 
points  early  in  the  series  which  were  equal  to  zero.  To 
determine  if  this  impacted  the  results,  the  TIMES  package 
was  used  to  produce  original  series,  ACF  and  PACF  plots 
for  all  sixty-four  data  points  and  for  a  fifty-one  point 
data  set  which  deleted  the  initial  series  of  zeros.  The 
resulting  ACF  and  PACF  plots  indicated  only  a  slight 
impact),  so  the  sixty-four  point  data  set  was  used  for 
all  further  model  testing. 

Analysis  of  the  ACF  plot  indicated  an  exponential 
decay  pattern  which  would  be  indicative  of  an  auto¬ 
regressive  process  (13:194)  (see  Appendix  D).  Models 
tested  were:  AR(2),  AR(3),  and  MA(3).  Of  these  simple 
models,  the  AR(2)  model  had  the  best  fit.  The  Portmanteau 
lack  of  fit  test  Q  statistic  was  9.8117  with  nine  degrees 
of  freedom.  This  was  substantially  lower  than  the  original 

series  Q  value  of  55.247  with  twelve  degrees  of  freedom 

2 

(see  Table  4.2),  but  was  still  greater  than  Xg  =  3.325. 

Both  the  periodogram  and  the  autoregression  function  still 
indicated  residual  autocorrelation  of  periods  five  and  six 
(see  Appendix  D).  The  next  model  tested  was  ARIMA 


(2,0,0)*(l,0,0)g  to  eliminate  the  significant  autocorre¬ 
lations  at  lags  five  and  ten.  Results  showed  improvement. 
The  Portmanteau  lack  of  fit  test  Q  statistic  was  7.1077 
with  eight  degrees  of  freedom.  This  was  a  substantial  drop 

in  Q  for  a  loss  of  only  one  degree  of  freedom.  However, 

2 

the  Q  statistic  7.1077  was  greater  than  Xg  =  2.733.  The 

ACF  and  PACF  plots  still  showed  strong  autocorrelation  at 

lag  six.  To  eliminate  this  autocorrelation,  the  model 

ARIMA  (2,0,0)*(l,0,0)g*(0,0,l)g  was  tested.  Results  were 

very  good  (see  Table  4.2).  The  Q  statistic  was  2.5391  with 

2 

seven  degrees  of  freedom  and  X7  =  2.167. 

The  final  model  for  the  F-16  data  is  an  ARIMA 
(2,0,0)*(l,0,0)5*(0,0,l)g  model.  The  model  equation 
(in  Box-Jenkins  backshift  operator  notation  is 

(1  -  . 37875B  -  . 25808B2 )  (1  -  .23910B5)  Zfc  = 

[4.4] 

(1  -  . 29997Bd )  et 

Multiplying  both  sides  of  equation  [4.4]  yields  the 
equat ion : 

Zt  -  .  37875BZ..  +  .25808B2zL  +  .  23910B5Z<. 

C  r  [4.5] 

-  .090559lB6Zt  -  .0617088B7Zt  +  6t  -  .29997B76t 

Equation  [4.5]  can  be  expressed  in  more  conventional 

notation  by  removing  the  backshift  operators  to  yield  the 

final  equation: 


Zt  =  .37875Zt_1  +  .25808Zt_2  +  .23910Zfc_5 

-  .0905591Zt_g  -  .0617088Zt_?  +  efc  [4.6] 

-  .29997et_g 

Throughout  the  modeling  process  for  both  data  sets, 
the  periodograms  and  spectrograms  were  analyzed  for 
in^rovements.  The  computer  diagnostics  for  the  initial 
run  and  final  model  for  both  data  sets  are  included  in 
Appendix  D. 

The  first  article  delivery  dates  were  November  1974 
(month  nine)  for  the  F-15  and  August  1978  (month  twelve) 
for  the  F-16.  These  dates  occurred  long  before  the  major 
portion  of  the  ECPs  impacting  logistics  were  generated. 
Combined  with  the  fact  that  each  system  has  a  different 
model,  the  first  article  delivery  dates  appear  to  have 
occurred  too  early  to  be  useful  in  predicting  design 
stability. 

Research  Question  Number  Two 

Research  question  number  two  stated,  "If  such  a  trend 
exists,  demonstrate  that  this  trend  can  be  applied  to  all 
like  systems,  so  that  it  can  be  used  as  a  predictor  of  the 
point  at  which  design  stability  will  be  achieved."  As 


stated  in  Chapter  III,  Research  Methodology,  the  time 
series  plots  for  the  F-15  and  F-16  data  sets  were  analyzed 


for  similarity.  The  models  developed  for  the  two  data  sets 
are  very  different  (as  evidenced  by  the  different 
periodicities),  and  indicate  that  a  trend  for  one  system 
cannot  be  assumed  to  be  similar  to  the  trend  for  any  other 
system.  Thus,  the  models  developed  in  this  thesis  ought 
not  to  be  used  as  predictors  of  trends  for  any  other  like 
systems. 

Research  Question  Number  Three 

Research  question  number  three  stated,  "If  there  is 
a  trend,  demonstrate  that  it  can  be  applied  to  those  ECPs 
which  have  a  direct  impact  on  logistics  support  element 
development."  As  stated  in  Chapter  III,  Research 
Methodology,  all  ECPs  used  in  this  thesis  effort  did 
inpact  on  logistics  elements.  This  effort  has  identified 
the  existence  of  a  trend  for  those  ECPs  which  impact  on 
logistics  elements.  However,  there  was  not  sufficient  time 
to  evaluate  total  airframe  ECPs  and  compare  those  trends 
and  models  to  the  trends  and  models  developed  here. 

Qualitative  Analysis 

The  question  in  the  interview  guide  were  designed 
to  provide  answers  which  would  help  to  evaluate  research 
question  number  four.  It  stated,  "Investigate  the  adequacy 
of  existing  methods  of  determining  whether  or  not  a  design 
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is  stable  for  purposes  of  developing  organic  logistics 
support  capability."  Responses  to  each  interview  question 
were  categorized  and  tallied  to  identify  similarities  and 
differences  among  them.  The  demographic  data  were  used  to 
ascertain  any  variations  among  responses  which  could  be 
attributed  to  amount  of  experience  managing  logistics 
support  and/or  ICS. 

An  analysis  of  the  responses  to  each  of  the  interview 
questions  follows.  Responses  to  the  demographic  questions 
and  to  interview  questions  one  through  nine  are  summarized 
in  Appendix  E.  Responses  to  interview  question  ten  are 
given  in  their  entirety  in  this  analysis  discussion. 

Interview  Question  Number  One 

Interview  question  number  one  asked,  "Design 
instability  is  often  cited  as  a  reason  for  using  interim 
contractor  support  (ICS).  How  would  you  define  design 
stability  vs.  instability  as  it  relates  to  such  a  decision, 
and  to  logistics  supportability  of  a  new/modified  weapon 
system?"  This  question  was  designed  to  obtain  an  idea  of 
what  logistics  managers  look  for  in  determining  design 
stability,  and  when  they  think  it  has  been  achieved. 

The  most  frequently  given  response  was  "Design 
stability  has  been  achieved  when  system  reliability  or  MTBF 
specifications  have  been  achieved."  Four  people  gave  that 


response.  Three  people  cited  "advancement  of  the  state  of 
the  art"  as  the  biggest  cause  of  design  instability.  Three 
people  identified  critical  design  review  (CDR)  as  the  point 
at  which  design  stability  is  achieved,  and  another  cited 
the  reliability  allocation,  assessment,  and  analysis  report 
(AAA  Report)  as  an  indicator.  It  is  interesting  to  note 
that  the  person  who  cited  the  AAA  Report  also  defined 
design  stability  in  terms  of  achieved  reliability 
specifications,  but  none  of  the  other  responses  mentioned 
were  from  the  same  individuals.  Three  individuals  stated 
that  achievement  of  design  stability  could  not  be  tied  to 
any  specific  program  milestone.  When  these  answers  were 
compared  to  the  demographic  data,  no  correlation  was  found 
between  amount  of  management  experience  and  any  of  the 
answers. 


Interview  Question  Number  Two 


Interview  question  number  two  asked,  "Do  any  of 
the  following  items  have  an  effect  on  the  date  at  which 
a  weapon  system  achieves  logistics  supportability?  If 
so,  please  indicate  whether  the  effect  is  significant  or 
minimal.  Include  any  additional  items  that  may  affect 
logistics  supportability  under  ’other'.  Please  rank 
according  to  impact  (1  =  most  impact,  etc.).  Ties  are 
allowed  if  necessary."  The  complete  list  of  items 
considered  is  included  in  Appendix  E. 
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Concurrency  of  FSD  and  production  programs  was  ranked 
first  in  impact  by  six  people  and  was  rated  as  significant 


by  eight.  It  was  rated  second  in  impact  by  only  one 
person.  Complexity  of  equipment  was  ranked  first  in  impact 
by  five  people  and  was  rated  as  significant  by  nine.  It 
was  ranked  second  in  impact  by  three  people.  Three  people 
added  funding  availability,  ranking  it  first  in  impact. 
Number  and  timeframe  of  ECPs,  IOC  date,  first  article 
delivery  date,  and  support  equipment  development  were  all 
ranked  first  by  three  people.  Support  equipment  develop¬ 
ment  was  ranked  second  by  three  people  also,  and  was  rated 
significant  by  nine.  IOC  date  was  considered  significant 
by  eight  people.  Seven  people  considered  number  and 
timeframe  of  ECPs  and  length  of  FSD  phase  to  be  signifi¬ 
cant.  Only  one  person  considered  concurrency  of  FSD  and 
production  programs  to  be  of  minimal  inpact,  and  two  people 
rated  complexity  of  equipment  as  minimal.  Other  responses 
were  varied.  There  was  no  correlation  between  amount  of 
experience  and  any  of  the  responses  to  the  question. 

* 

Interview  Question  Number  Three 

Interview  question  number  three  asked,  "What  is  the 
effect  on  the  following  of  a  delay  in  reaching  design 
stability?  Indicate  significant,  minimal,  or  none." 
Responses  were  requested  for  delays  of  from  one  to  three 


months  and  for  delays  of  greater  than  three  months. 

The  complete  list  of  elements  considered  is  given  in 
Appendix  E. 

Delays  of  from  one  to  three  months.  Five  people 
considered  the  effect  on  support  equipment  delivery,  ICS, 
and  organic  capability  date  to  be  significant  for  delays 
of  from  one  to  three  months.  Four  people  considered  the 
impact  on  LCC  and  spares  provisioning  to  be  significant. 
Most  people  considered  delays  of  this  length  to  have 
minimal  effect  on  the  elements  listed.  Very  few  believed 
there  would  be  no  impact  at  all. 

Delays  of  greater  than  three  months.  A  majority 
of  people  considered  delays  of  this  length  to  have  a 
significant  impact  on  all  elements  listed.  All  eleven 
people  interviewed  agreed  that  a  delay  of  three  months 
or  more  would  significantly  impact  the  organic  capability 
date.  Ten  people  believed  the  delay  would  significantly 
impact  support  equipment  delivery  and  technical  document 
preparation.  Nine  people  felt  the  delay  would  be  signifi¬ 
cant  to  support  equipment  development,  spares  provisioning 
computations  and  spares  delivery  dates.  Eight  people  said 
that  ICS  and  training  equipment  development  would  be 
impacted  significantly.  Seven  people  thought  the  delay 
would  significantly  impact  support  equipment  requirement 
(SERD)  development  and  LCC.  Five  people  believed  the 
impact  on  training  development  would  be  significant.  Only 


three  people  felt  that  LSA  would  be  significantly  impacted, 
and  four  felt  that  the  impact  would  be  minimal.  Two  said 
there  would  not  be  any  impact  on  LSA  or  SERDs.  Two  people 
believed  the  effects  on  LCC  would  be  minimal.  There  was  a 
consensus  of  opinion  among  the  respondents  that  the  greater 
the  length  of  the  delay,  the  greater  the  significance  of 
the  impact.  Experience  level  was  not  correlated  with  the 
responses  to  this  question. 


Interview  Question  Number  Four 

Interview  question  number  four  stated,  "One  technique 
currently  used  to  assess  design  stability  is  a  trend 
analysis  of  engineering  change  proposals  (ECPs).  Please 
identify  any  other  techniques  which  you  have  used  (or  are 
aware  of)  for  assessing  the  design  stability  of  a  weapon 
system  from  a  logistics  supportability  viewpoint." 

Responses  to  this  question  were  quite  varied.  The 
largest  number  of  similar  responses  was  three.  Those 
three  people  cited  the  use  of  reliability  analyses  and  the 
achievement  of  a  mature  MTBF  (specification  MTBF).  In  all, 
nineteen  techniques  were  mentioned.  One  individual  cited 
seven,  and  two  people  claimed  not  to  be  aware  of  any. 

There  appear  to  be  quite  a  lot  of  different  techniques  in 
use  for  assessing  design  stability.  Once  again,  experience 
level  did  not  have  any  obvious  impact  on  the  responses. 


Interview  Question  Number  Five 

Interview  question  number  five  asked,  "Prior  to 
making  a  decision  to  use  ICS,  AFR  800-21  requires  'rigorous 
cost  and  risk  analyses'  to  support  such  a  decision.  Can 
you  comment  on  the  adequacy  of  techniques  available  to 
perform  such  analyses?" 

Five  respondents  stated  that  the  available  techniques 
were  adequate.  Four  of  the  five  had  over  fourteen  years  of 
logistics  management  experience.  Two  of  them  stated  that 
techniques  must  be  adequate  since  decisions  to  use  ICS 
were  seldom  found  to  be  incorrect  after  the  fact.  Four 
people  stated  they  thought  the  techniques  available  were 
inadequate.  All  four  had  six  or  fewer  years  of  logistics 
management  experience.  Perhaps  experience  influences  the 
perceptions  about  the  adequacy  of  the  available  techniques, 
or  the  ability  to  interpret  their  results.  There  was  no 
consensus  regarding  adequacy  of  techniques  available.  In 
fact,  people  often  gave  opposite  responses.  For  example, 
AFALD  was  credited  with  being  of  great  assistance  to  one 
individual  and  also  cited  for  giving  inadequate  and 
irrelevant  guidance  to  another. 

Interview  Question  Number  Six 

Interview  question  number  six  asked,  "What  techniques 


have  you  employed  in  the  past  to  perform  such  analyses?" 


Four  people  said  they  had  used  cost  analyses,  and 


three  said  they  used  pre-planning  for  ICS  in  lieu  of  such 
an  analyses.  One  used  both  cost  analyses  and  pre-planning 
for  ICS.  Two  people  gave  four  responses,  one  gave  five, 
and  one  gave  six.  All  others  gave  only  one  or  two.  Only  a 
few  individuals  appeared  to  be  attempting  very  many  methods 
of  performing  such  analyses.  Three  people  had  used  trade 
studies  of  ICS  vs.  100%  NRTS.  Experience  level  was  not 
correlated  with  these  responses. 

Interview  Question  Number  Seven 

Interview  question  number  seven  stated,  "Please  list 
all  such  techniques  with  which  you  are  familiar  and  rank 
them  according  to  how  effective  you  believe  them  to  be." 

Three  people  ranked  ICS  vs.  100%  NRTS  first.  Two  of 
the  three  had  listed  five  and  six  techniques  for  question 
number  six.  The  three  people  were  also  the  only  ones  who 
had  used  that  technique.  LSA  was  ranked  first  by  two 
people.  No  other  methods  received  any  consensus,  and 
experience  level  of  the  respondents  was  not  correlated  with 
the  rankings. 

Interview  Question  Number  Eight 

Interview  question  number  eight  asked,  "The  previous 


seven  questions  have  addressed  the  point  of  design 


stability  vs.  instability  as  it  relates  to  decisions 
about  the  use  of  ICS.  Would  you  care  to  comment  on  the 
feasibility  of  establishing  such  a  point,  and  on  its 
usefulness  in  making  such  decisions?" 

Three  people  stated  they  felt  design  stability  was 
over-stressed.  One  of  these  people  had  forty  years  of 
logistics  management  experience  and  had  participated  in  the 
preparation  of  AFR  800-21.  That  individual  commented  that 
when  the  regulation  was  written,  no  one  foresaw  the  effect 
that  the  inclusion  of  situation  one  would  have,  or  the 
confusion  it  would  create,  and  recommended  its  deletion 
from  the  regulation.  Three  different  people  stated  that 
pre-planning  for  ICS  was  necessary  and  that  such 
pre-planning  precluded  the  necessity  of  identifying  design 
stability.  Two  of  those  people,  plus  a  third,  stated  that 
even  when  a  point  was  determined,  it  was  usually  too  late 
to  allow  for  cost-effective  ICS  planning.  Three  different 
individuals  stressed  that  lack  of  support  resources  was  the 
real  driver,  but  one  of  them  felt  that  design  instability 
drives  the  lack  of  support  resources.  Two  individuals  felt 
that  a  contractually  defined  design  stability  point  could 
be  useful  in  providing  the  contractor  with  incentives  to 
achieve  stability.  Again,  there  did  not  appear  to  be  any 
correlation  between  experience  level  and  the  responses  to 
this  question. 


Interview  Question  Number  Nine 

Interview  question  number  nine  asked,  "For  programs 
with  which  you  are  familiar,  which  of  the  two  situations 
described  in  AFR  800-21  resulted  in  a  requirement  for  ICS?" 

A  large  majority  of  justifications  for  the  use  of  ICS 
resulted  from  only  situation  two.  A  few  were  driven  by 
both  situations.  One  individual  stated  that  a  certain 
program  was  "almost  a  one."  None  of  the  programs  used  only 
situation  one  as  the  justification  for  using  ICS. 


Interview  Question  Number  Ten 

Interview  question  number  ten  asked,  "Do  you  have  any 
additional  comments  on  this  subject?" 

Responses  to  this  question  were  so  varied  that  no 
real  consensus  is  evident.  Many  did  not  directly  address 
the  specific  issue  of  design  stability  addressed  in  this 
thesis,  but  were  related.  All  reflected  the  specific  areas 
in  which  a  particular  respondent  had  experienced  problems 
while  planning  for  logistics  support  of  a  weapon  system. 

The  comments  are  included  here  to  provide  a  basis  for 
further  discussion  and  recommendations. 

—  Yes.  As  one  of  the  original  contributors  to  the 
development  of  the  reg  [sic],  I  think  design  stability  as 
a  requirement  was  overstated.  The  inevitable  result  of 
design  instability  is  a  deferred  rescheduling  of  support 
resources,  hence  lack  of  support  resources  is  the  real 
driver  and  could  be  a  sufficient  reason.  This  was  not 
foreseen  at  the  time,  but  the  evidence  is  now  in.  There 


is  no  mystery  to  the  need  for  ICS — hence  it  should  always 
be  planned  for! 

—  Concurrency  is  the  biggest  challenge  to  logistics. 

—  If  concurrent ,  it  is  a  calculated  risk  going  ahead 
with  the  program  if  [the  system]  is  not  design  stable. 
Design  immaturity  paralleling  production  leads  to  support 
resources  not  being  available. 

—  If  FSD  is  compressed,  that  compresses  the  logistics 
support  identification. 

—  Even  though  the  design  is  unstable,  we  can  begin 
fundamental  ILS  planning  (i.e.,  inputs/outputs  to  be 
measured ) . 

—  Support  equipment  support  is  as  complex  as  PME 
support . 

—  After  CDR,  changes  equal  dollars. 

—  You  cannot  look  at  ICS  in  isolation;  you  must  view 
it  within  the  whole  acquisition  program.  The  more  ICS  is 
planned  for,  the  better  is  the  management  which  is  being 
used . 

—  The  eiqphasis  should  be  on  pre-planning.  The  prime 
objective  is  to  field  a  supportable  system.  To  the  extent 
that  you  have  to  use  ICS  to  do  that ,  then  that ' s  what 
you'll  have  to  do. 

—  The  smart  manager  plans  for  the  "unks"  [unknowns]. 

—  Key:  Loggies  [sic]  have  got  to  be  there  early  in 
program  planning.  The  key  to  LSA  is  early  up-front  plans 
and  studies  looked  at  to  make  meaningful  decisions  about 
long-range  implications  of  logistics. 

—  Proactive  not  reactive  management. 

—  Make  logistics  important  in  source  selection. 

—  Recognize  [the  need]  early;  include  ICS  as  a  contract 
option.  [This]  can  negate  the  need  for  ICS  or  result  in 
early  transition. 

—  Always  include  incentives  to  get  out  of  ICS. 

—  Motivate  the  contractor  not  to  want  to  go  into  ICS, 
but  have  it  as  an  option.  Then  it's  covered  if  necessary. 


We  used  to  make  decisions  because  funds  were  not 
available.  Now  we're  saying  even  if  funds  are  available, 
what's  the  better  way  of  going?  And  that's  a  smarter 
management  decision. 

—  Define  ICS  as  a  capability. 

—  Leadtime  for  spares  deliveries  and  software 
programming  are  the  biggest  drivers  for  ICS.  Technical 
data  (i.e.,  depot  level  technical  orders)  is  the  next 
long  lead  item.  Spares  should  not  matter  for  ICS  or 
organic  but  because  they  are  required  to  accomplish 
software  development,  they  become  a  driver.  With  a 
leadtime  of  thirty  to  thirty-six  months  for  software 
development,  that  would  require  a  finalized  software 
program  three  years  prior  to  delivery  of  the  first 
aircraft.  Our  configuration  is  still  changing  1}  [sic] 
years  after  delivery  of  the  first  aircraft.  It  is  obvious 
we  could  not  have  ordered  software  4i  [sic]  years  ago. 

—  Lack  of  time  to  develop  support  resources  is  the  main 
reason  for  using  ICS,  sometimes  due  to  a  late  CDR  date,  but 
not  necessarily  a  requirement.  ICS  should  be  seen  as  a 
viable,  acceptable  alternative  for  the  Air  Force  in  support 
planning,  rather  than  a  catch-up/band -a id  admittance  of 
failure,  as  it  is  currently  perceived.  Accelerated 
development/acquisition  and  restricted  funding  force  a 
program  into  the  ICS  arena.  However,  since  ICS  is  funded 
separately,  through  the  AFLC  chain,  it  is  not  always  a  fair 
tradeoff  when  considering  budget  cuts  in  programs.  If  the 
ICS  costs  were  part  of  the  overall  program  costs,  and  if 
the  program  manager  had  control  of  these  and  all  support 
dollars  (spares),  a  better  cost  analysis  could  be  done. 

—  A  serious  problem  which  is  tied  very  closely  to  ICS 
is  the  provisioning  area.  Spares  are  computed  for  organic 
repair  cycles  only.  These  quantities  are  not  sufficient 
for  ICS  repairs.  Also,  spare  funding  limitations  and 
restrictions  (design  stability)  are  critical  factors.  In 
a  concurrent  program,  the  provisioning  reg's  [sic]  should 
be  altered  to  permit  accelerated  procurement,  thus, 
reducing  the  cost  of  ICS — at  the  expense  of  the  gov't  [sic] 
assuming  the  risk  of  obsolete  spares.  These  cost  analyses 
are  not  being  done  currently. 

—  ICS  should  be  avoided  whenever  possible.  My  exper¬ 
ience  shows  that  it  is  much  too  expensive.  Perhaps  the 
Systems  Command  program  managers  should  pay  more  attention 
to  the  "rigorous  cost  analyses."  Also,  in  those  instances 
where  design  instability  is  the  justification,  perhaps  the 
system  shouldn't  be  deployed. 

—  FCA  and  PCA  are  not  significant. 


CHAPTER  V 


SUMMARY,  RECOMMENDATIONS  AND  CONCLUSIONS 

Summary 

This  thesis  attempted  to  determine  the  feasibility 
of  identifying  a  point  in  time  at  which  a  weapon  system 
achieves  design  stability,  and  of  using  that  point  when 
making  logistics  supportability  decisions.  The  analysis 
was  conducted  in  two  parts:  1)  a  quantitative  trend 
analysis  of  ECPs,  and  2)  a  qualitative  interview  series. 
Results  of  both  analyses  lead  to  the  conclusion  that 
identification  of  such  a  point  may  not  be  possible. 

Results  of  the  ECP  trend  analysis  indicate  that  while 
time  series  analysis  can  be  used  to  define  a  model  for 
a  given  set  of  historical  data,  the  models  for  two  very 
similar  systems,  the  F-15  and  F-16  fire  control  and  flight 
control  systems,  are  entirely  different.  Thus,  one  cannot 
use  a  model  developed  on  the  basis  of  historical  data  from 
one  program  to  predict  the  ECP  trend  for  a  new  weapon 
system  program.  Since  the  models  developed  in  this  thesis 
were  developed  using  only  those  ECPs  which  impacted 
logistics  elements,  it  is  apparent  that  the  effect  of  ECPs 
on  the  development  of  a  logistics  support  capability  cannot 
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be  said  to  be  the  same  for  any  two  given  weapon  systems. 

The  utility  of  such  models  to  logistics  managers  is 
questionable. 

Responses  to  the  interview  questions  indicate  that 
while  some  of  the  subjects  believe  that  design  stability 
can  be  measured,  few  agreed  on  how  or  on  the  definition  of 
design  stability  itself.  While  most  agreed  that  design 
stability  can  and  does  impact  logistics  support  capability 
and  interim  contractor  support,  they  felt  that  it  occurs 
too  late  in  a  weapons  system's  life  cycle  to  be  a  cost- 
effective  decision  parameter.  They  believed  the  real 
driver  in  the  logistics  decision-making  process  to  be 
support  resources  availability  and  lead  times.  The  most 
commonly  cited  factors  impacting  design  stability  included 
achieved  MTBF,  state  of  the  art  technology,  and  concurrency 
of  FSD  and  production  programs.  Those  factors  differ  for 
every  program. 

Although  the  interview  sample  population  was  not  a 
statistically  significant  sample,  the  subjects  were  drawn 
from  a  population  of  people  actively  involved  in  making 
decisions  about  weapon  system  supportability.  Their 
responses  were  based  on  their  experience  with  the  acquisi¬ 
tion  of  logistics  support  resources  and  ICS,  following  the 
current  DOD  and  Air  Force  regulations.  Their  responses 
indicate  that  they  don't  use  design  stability  as  a  decision 
parameter  when  making  logistics  supportability  decisions. 


They  rely  on  other  means  of  determining  their  requirements. 
Responses  to  interview  question  nine  indicate  that  Situa¬ 
tion  1,  design  instability,  was  not  used  as  a  justification 
for  ICS  by  any  of  the  interview  subjects  on  any  of  their 
programs.  These  findings  are  consistent  with  the  AIA  Study 
conclusion  that,  "The  results  of  this  questionnaire  .  .  . 
did  not  surface  any  substantial  new  data  that  could  be  used 
to  develop  criteria  for  determining  when  a  weapon  system 
has  achieved  design  stability,"  and  "In  view  of  the  minimal 
results  .  .  .  further  pursuit  of  this  subject  does  not 
appear  to  be  productive.  [1]" 

The  results  of  the  ECP  trend  analysis  portion  of 
this  thesis  support  the  conclusion  of  the  ARINC  study  that 
"the  number  of  ECPs  submitted  by  the  prime  contractor 
approaches  zero  as  the  end  of  production  is  reached 
[2s 131."  A  second  conclusion  of  that  study,  that  "The 
number  of  ECPs  being  submitted  each  year  on  the  P-3C  is 
still  increasing,  particularly  for  the  avionic  subsystems, 
possibly  reflecting  their  complexity  [2:131"  is  supported 
by  the  responses  to  the  interview  portion  of  this  thesis. 
Many  responses  indicated  continued  ECP  generation  after 
first  article  delivery.  The  ARINC  study  recommended  that 
a  similar  study  be  conducted  to  "verify  the  results  of  the 
P-3  ECP  Trend  Analysis  and  to  permit  these  results  to  be 
extended  to  other  aircraft  types  [2; 131."  The  Bemis  study 
"indicated  that  the  shape  of  the  engineering  change  traffic 


profile  was  of  a  similar  shape  for  different  kinds  of 
systems  [3]."  Bemis'  finding  was  based  on  the  use  of 
regression  analysis ,  and  was  not  replicated  by  this  thesis 
effort.  This  effort  verified  the  existence  of  a  trend  in 
the  rate  of  generation  of  ECPs  for  two  similar  fighter 
aircraft  avionics  systems  (F-15  and  F-16  fire  control  and 
flight  control  systems).  However,  each  aircraft  system 
exhibited  its  own  distinct  trend,  and  neither  can  be 
extended  to  other  aircraft  systems.  This  thesis  used 
Box-Jenkins  time  series  analysis,  since  the  ECP  data 
indicated  the  presence  of  autocorrelation.  The  use  of 
multiple  regression  was  therefore  contraindicated.  Any 
further  efforts  in  this  area  should  use  Box-Jenkins  time 
series  analysis. 

The  qualitative  portion  of  this  thesis  effort 
supported  the  results  of  the  AIA  Study.  Responses  to 
the  AIA  questionnaire  indicated  that  "the  intact  of  any 
delays  in  achieving  design  stability  on  the  development 
of  logistics  elements  is  significant  [1]."  Responses 
to  interview  questions  two  and  three  lead  to  the  same 
conclusion.  The  AIA  Study  recommended 


effective  planning  and  implement at ion  of  Interim 
Contractor  Support  ...  as  the  best  approach  to 
provide  logistics  support  services  during  periods 
of  design  instability  and  the  time  period  needed  to 
establish  organic  support  capability  Cl]. 
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Many  of  the  responses  to  the  thesis  interview  questions 
support  the  pre-planned  use  of  ICS  as  a  contract  option  to 
be  exercised  if  needed.  The  Air  Force  Audit  Agency  Report 
stated  that,  "each  program  has  unique  considerations  that 
must  be  analyzed  on  a  case-by-case  basis  [20:9-10].” 

The  interview  responses  indicate  that  many  of  the  subjects 
believe  that  a  pre-planned  ICS  option  can  be  used  in  lieu 
of  (or  negates  the  need  for)  the  "rigorous  cost  and  risk 
analyses"  required  by  AFR  800-21.  The  Audit  Agency  Report 
reiterates  the  requirement  to  perform  such  analyses,  and 
cites  the  "absence  of  coaqplete  directive  guidance  [20:9]" 
as  a  factor  contributing  to  such  incomplete  planning. 

Responses  to  the  interview  questions  indicate  that 
many  logistics  managers  believe  that  adequate  techniques 
exist  to  perform  such  analyses.  A  few,  however,  felt  that 
adequate  guidance  was  lacking.  The  current  tendency  in 
acquisition  toward  concurrent  development  and  production 
programs  and  the  propensity  to  advance  the  state  of  the 
art  were  often  cited  as  reasons  why  pre-planned  ICS  is 
a  necessity.  Such  situations  usually  result  in  long 
lead  times  for  support  resource  development.  Former 
Deputy  Secretary  of  Defense  Frank  C.  Carlucci  established 
thirty-two  initiatives  for  streamlining  the  weapon 
system  acquisition  process.  Recommendation  16  addresses 
"Contractor  Incentives  to  Improve  Reliability  and  Support" 
(6:18).  Current  regulations  and  guidance  are  not  entirely 


consistent  with  this  and  many  of  his  other  recommendations, 
resulting  in  many  problems  in  interpretation  and  implemen¬ 
tation.  The  regulations  and  guidance  should  be  revised 
to  reflect  current  acquisition  policies  and  practices, 
thereby  enabling  logistics  managers  to  accomplish  adequate 
planning. 

Two  specific  areas  of  concern  to  many  of  the 
interview  subjects  are  spares  support  and  software  support 
requirements  planning.  AFR  800-21  provides  little  guidance 
in  these  areas.  Since  ICS  is  a  special  type  of  support 
and  differs  from  organic  support  in  many  ways,  spares 
provisioning  based  on  organic  maintenance  often  is  not 
adequate  for  ICS. 

Recommendations 

1.  It  is  recommended  that  Situation  1,  design 
instability,  be  considered  for  deletion  from  AFR  800-21 
as  a  justification  for  the  use  of  ICS.  This  recommendation 
is  based  on  the  results  of  the  ECP  trend  analysis  which 
demonstrates  that  a  practical  quantitative  technique  for 
forecasting  design  stability  may  not  exist.  This  result 
is  coupled  with  the  lack  of  consensus  among  the  interview 
subjects  as  to  how  to  define  design  stability,  when  it 
occurs,  and  its  usefulness  as  a  decision  parameter  for 
logistics  supportability  decisions.  The  fact  that  none  of 


the  interview  subjects  has  ever  used  Situation  1  provides 
further  support  for  this  recommendation. 

2.  It  is  recommended  that  an  attempt  be  made  to 
replicate  the  results  of  this  thesis  using  Box-Jenkins  time 
series  analysis  on  similar  data.  Since  neither  the  Bemis 
study  nor  the  ARINC  study  employed  Box-Jenkins  time  series 
analysis,  results  are  not  strictly  comparable.  An 
additional  study  of  this  type  is  needed  to  confirm  the 
results  of  this  thesis. 

3.  It  is  recommended  that  further  study  be 
accomplished  to  determine  adequacy  of  available  analysis 
techniques  for  accomplishing  the  cost  and  risk  analyses 
required  by  AFR  800-21,  and  that  the  regulation  be  revised 
to  provide  more  explicit  definition  and  explanation  of  such 
techniques.  (Rigid  requirements  are  not  desirable,  and  are 
not  recommended. ) 

4.  It  is  recommended  that  further  study  be 
accomplished  in  the  areas  of  spares  support  and  software 
support  requirements  planning,  as  related  to  the  use  of 
ICS.  Guidance  in  these  areas  is  minimal,  and  needs  to  be 
revised  and  expanded.  (Again,  rigid  requirements  are  not 
desirable,  and  are  not  recommended.) 
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Conclusions 


This  research  effort  has  determined  that  design 
stability  is  not  a  very  useful  decision  parameter  for 
making  logistics  supportability  decisions.  Although  it 
has  an  impact  on  all  logistics  elements,  it  cannot  be 
adequately  defined  and  is  so  program  specific  as  to  be 
virtually  unpredictable.  ECP  trend  analysis  can  be  used 
on  historical  data  to  develop  a  model,  but  the  model  cannot 
be  used  to  predict  for  a  new  system.  Logistics  managers 
currently  involved  in  the  acquisition  of  support  resources 
for  Air  Force  weapon  systems  are  using  other  methods  and 
parameters  in  their  planning  processes.  Emphasis  should  be 
placed  on  the  planning  effort,  and  ICS  should  be  used  as  an 
option  when  support  resources  are  not  available  when 
required  in  the  field. 
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APPENDIX  A 

TERMS  AND  ACRONYMS  DEFINED 


Class  V  Modification.  Modification  of  a  system  or  equip¬ 
ment  that  will  provide:  1)  a  change  in  operational 
requirements  or  performance  which  provides  an 
added  capability  not  inherent  in  the  baseline 
configuration;  2)  the  capability  to  accomplish  an 
assigned  mission  that  the  basic  system  or  equipment 
was  not  originally  designed  to  accomplish;  3)  a 
significant  and  measurable  training  or  logistic 
improvement  certified  essential  by  the  command  or 
the  agency  primarily  concerned;  4)  a  modification 
required  to  improve  system  operational  capability 
(change  in  mission)  [17:123]. 

Critical  Design  Review  (CDR ) .  A  detailed  design  specifi¬ 
cation  review  conducted  at  the  end  of  FSD  prior  to 
the  production  decision  to  verify  the  adequacy  and 
producibility  of  the  design.  Design  is  frozen  at 
CDR. 

Deputy  Program  Manager  for  Logistics  (DPML ) .  An  exper¬ 
ienced  logistician  assigned  by  AFLC  to  a  Program 
Office  to  serve  as  one  of  the  deputies  on  the  Program 
Manager's  (PM)  staff.  Responsible  for  logistics 
support  planning  and  acquisition  during  the  weapon 
system  acquisition  process  [17:218]. 

Development  Test  and  Evaluation  (DT&E) .  Test  and  evalu¬ 
ation  conducted  to  demonstrate  that  engineering 
design  and  development  are  complete  and  that  the 
system  or  equipment  meets  specifications.  DT&E  also 
verifies  that  proposed  design  changes  do  not  degrade 
overall  system  performance.  DT&E  is  conducted  by 
the  implementing  command. 

Follow-on  Test  and  Evaluation  (FOT&E).  Test  and  evalu¬ 
ation  which  is  conducted  after  IOT&E  to  continue 
and  refine  the  estimates  made  during  IOT&E,  to 
evaluate  changes,  and  to  reevaluate  the  system  to 
insure  that  it  continues  to  meet  operational  needs 
and  retain  its  effectiveness  in  a  new  environment 
or  against  a  new  threat. 

Initial  Operational  Test  and  Evaluation  (IOT&E).  That 

portion  of  operational  test  and  evaluation  conducted 
prior  to  the  first  major  production  decision.  The 
system  is  evaluated  against  operational  criteria  by 
personnel  with  the  same  skills  and  equipment  which 
will  be  used  in  the  operational  environment. 
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Integrated  Logistics  Support  Plan  (ILSP).  A  document 

which  provides  a  comprehensive  and  detailed  plan  for 
implementing  the  concepts,  techniques,  and  policies 
necessary  to  achieve  the  integrated  logistics  support 
(ILS)  objectives  of  assuring  the  effective  economical 
support  of  a  system  or  equipment  for  its  life  cycle 
(AFR  800-8)  [17-356]. 

Justification  of  Major  System  New  Start  (JMSNS).  Justifies 
the  need  for  a  major  system  new  start  to  fill  a 
mission  requirement  or  meet  a  threat. 

Life  Cycle  Cost  (LCC).  The  total  cost  of  an  item  or  system 
over  its  full  life.  It  includes  the  cost  of  develop¬ 
ment,  production,  ownership  (operation,  maintenance, 
support,  etc.)  and,  where  applicable,  disposal 
[17:390]. 

Maintainability.  A  characteristic  of  design  and  instal¬ 
lation  expressed  as  the  probability  that  an  item  will 
be  restored  to  a  specified  condition  within  a  given 
period  of  time  when  the  maintenance  is  performed 
using  prescribed  procedures  and  resources.  System 
maintainability  may  also  be  expressed  in  such  terms 
as  Mean-Time-to-Repair ,  Maintenance  Manhours  per 
Flying  Hour,  or  Mean  Down-Time  (AFR  80-14)  [17:406]. 

Maintenance  Engineering  Analysis  (MEA).  Similar  to  LSA; 

performs  a  design  to  cost,  life  cycle  cost  analysis. 
Superceded  by  LSA. 

Maintenance  Level  Analysis  (MLA).  Analysis  leading  to 
decisions  about  the  use  of  various  maintenance 
concepts  relative  to  reliability  impacts,  costs, 
and  other  tradeoffs. 

Not  Reparable  This  Station  ( NRTS ) .  The  percentage  of 

failed  items  which  must  be  sent  to  a  central  repair 
activity  having  greater  repair  capability,  or  not 
authorized  for  repair  at  that  location  [17:479]. 

Operational  performance.  Performance  standards  for 

operational  use  of  a  system,  stated  in  measurable 
terms . 

Preliminary  Design  Review  (PDR ) .  Equipment  design 
review  during  the  detail  design  development 
phase.  Engineering  models  are  tested  for  function, 
reliability,  maintainability,  etc.,  prior  to 
finalizing  the  detail  design. 


Producibility.  The  composite  of  characteristics  which, 
when  applied  to  equipment  design  and  production 
planning,  leads  to  the  most  effective  and  economic 
means  of  fabrication,  assembly,  inspection,  test, 
installation,  checkout,  and  acceptance  of  systems 
and  equipment  ( AFSCM  84-3)  [17:547], 

Production  Reliability  Qualification  Test  ( PRQT ) . 

Tests  conducted  on  the  system  or  equipment  prior 
to  production  go-ahead  to  evaluate  the  system 
development  progress  and  to  ensure  that  reliability 
specifications  have  been  met.  Also  evaluates  system 
supportability  and  compatibility  between  prime 
equipment  and  recommended  test  and  support  equipment. 

Program  Management  Responsibility  Transfer  ( PMRT ) .  The 

transfer  of  program  management  responsibility  for  a 
system  (by  series)  or  equipment  (by  designation), 
from  the  implementing  command  to  the  supporting 
command.  PMRT  includes  transfer  of  engineering 
responsibility  [17:555]. 

Reliability.  The  probability  that  a  system,  subsystem, 

component,  or  part  will  perform  a  required  function 
under  specified  conditions,  without  failure  for  a 
specified  period  of  time  ( AFLCM  72-2)  [17:576]. 

Reliability  Allocation  Assessment  and  Analysis  Report  (AAA 
Report).  This  report  is  used  to  1)  evaluate  the 
contractor's  estimate  of  reliability;  2)  evaluate 
the  potential  reliability  of  the  configuration  item 
design;  3)  provide  information  to  assist  in  directing 
and  planning  for  reliability  and  related  program 
efforts;  and  4)  identify  design  features  which  are 
critical  to  reliability. 

Repair  Level  Analysis  (RLA).  The  basic  decisions  about: 

1)  Repair  vs.  throwaway;  2)  the  most  desirable 
repair  posture  for  reparable  units  to  the  lowest 
level  between  a  single  point  depot  repair  capability 
and  multiple  base  levels  [17:580], 

Rough  Order  of  Magnitude  (ROM).  A  contractor's  rough 
estimate  of  cost  for  a  given  service  or  piece  of 
equipment.  Not  a  detailed  cost  breakdown,  but  a 
top-level  cost  only.  Not  contractually  binding. 
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DEPARTMENT  OF  THE  AIR  FORCE 
AIR  FORCE  INSTITUTE  OF  TECHNOLOGY  (ATC) 
WRIGHTPATTERSON  AIR  FORCE  SASE.  OH  45433 


13JUN  1983 


wmc?  Interim  Contractor  Support/Design  Stability  Study 


•3  HQ  AFALD/SDF  SDB  SDL  SDS  SDA  SDR 

1.  AFR  800-21  (26  September  1978)  defines  interim  contractor  support  (ICS)  as 
"a  cost-effective  logistics  support  alternative"  which  "allows  the  Air  Force 
to  defer  investment  in  all  or  part  of  the  support  -esources  .  .  .  and  to  use 
contractor  support  while  Che  organic  capability  is  being  phased  in."  ICS 
refers  only  to  the  use  of  a  contractor  in  two  situations:  (a)  The  items  tc 
be  supported  or  che  items  of  support  equipment  have  an  unstable  design; 

(b)  All  or  part  of  the  resources  required  to  establish  an  organic  capability 
will  not  be  available  until  after  operational  support  is  first  required.  For 
contractor  support  to  be  described  as  ICS  in  either  situation,  it  must  have 
been  subjected  to  rigorous  cost  and  risk  analyses. 

2.  Recent  government  and  Industry  studies  have  determined  that  design  stability 
from  a  logistics  supportablllcy  standpoint  is  relatively  undefined,  and  that 
coat  and  risk  analyses  as  required  by  AFR  800-21  are  noc  well  documented  by 
program  offices.  Ms.  Debra  Good,  a  graduate  student  in  the  School  of  Systems 
and  Logistics,  AFIT,  is  preparing  a  thesis  on  the  topic  of  design  stability  as 
it  relates  to  logistics  supportablllcy  decisions.  She  will  contact  you  within 
che  next  week  to  arrange  an  interview  at  your  convenience.  Your  thoughtful 
advance  preparation  of  the  attached  interview  guide  will  facilitate  the  inter¬ 
view.  Responses  will  be  kept  confidential,  and  names  will  noc  be  used  in  data 
analyses  or  conclusions. 

3.  Prior  to  attending  AFIT,  Ms.  Good  managed  ICS  for  the  8-52  Offensive  Avionics 
System.  After  completing  the  graduate  program  at  AFIT,  she  will  return  to 
AFALD/SD,  where  she  has  been  a  logistics  manager  for  five  years.  She  has  che 
full  cooperation  and  support  of  AFALD/SD  in  her  thesis  effort.  Your  cooperation 
will  be  greatly  appreciated  by  Ms.  Good,  AFIT,  and  AFALD/SD. 


I  Atch 

Questionnaire 


1st  Ind,  AFALD/SD  13  June  1983 

I  heartily  endorse  this  thesis  effort,  and  request  your  full  cooperation 
and  support. 

ROBERT  L.  OWEN 

Assistant  Deputy  for  Aeronautical  Programs 
Air  Force  Acquisition  Logistics  Division 

V*.  V  '.•« 


JEROME  G.  PE$_ 

Associate  Dean 
School  of  Systems  and  Logistics 
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INTERVIEW  GUIDE  FOR  DESIGN  STABILITY  STUDY 


According  to  AFR  800-21,  ICS  has  a  specific  meaning  and  refers  only  to  the  use  of 
a  contractor  in  the  two  situations  described  below. 

a.  Situation  1.  The  items  to  be  supported  or  items  of  support  equipment  have 
an  unstable  design;  moreover,  the  projected  cost  of  setting  up  an  organic  capabil¬ 
ity  at  the  time  operational  support  is  first  required  is  excessive,  either  because 
of  uncertainties  in  the  type  and  level  of  support  required,  or  because  of  the  risk 
that  support  resources  will  become  obsolete  if  procured  too  early.  For  contractor 
support  to  be  described  as  ICS  in  this  situation,  it  must  have  been  planned  at 
least  budget  lead  time  away  and  have  been  subjected  to  rigorous  cost  and  risk  ana¬ 
lyses. 


b.  Situation  2.  All  or  part  of  the  resources  required  to  establish  an  organic 
capability  will  not  be  available  until  after  operational  support  is  first  required. 
In  this  situation,  the  system  development,  production,  and  deployment  phases  do 
not  allow  enough  time  to  develop  the  support  resources  before  organic  support  is 
needed.  (The  most  common  example  of  this  situation  is  the  sequential  phasing  of 
automatic  test  equipment  development,  which  generally  lags  development  of  the 
system  it  supports.)  In  Situation  2,  as  in  Situation  1,  to  qualify  as  ICS,  early 
planning  and  analyses  must  occur. 

Under  the  ICS  concept,  the  Air  Force  conducts  risk  analyses  and  other  evaluations 
through  the  logistics  support  analysis  (LSA)  process,  to  identify  potentially 
unstable  design  items  and  items  having  potentially  high  initial  support  costs. 
Through  these  analyses,  it  identifies  those  areas  where  it  may  be  beneficial  to 
use  a  contractor  for  logistics  support  during  the  early  deployment  phase.  The 
Air  Force  then  selects  and  plans  the  most  feasible  and  practical  alternative  for 
providing  responsive  logistics  support  for  the  item  when  it  is  first  introduced 
into  the  operational  force. 

Risk  analysis  is  "an  analysis  that  evaluates  expected  performance  (cost,  schedule, 
or  technical)  as  compared  with  desired  performance,  with  a  view  toward  determining 
the  probability  that  requirements  will  be  met  within  available  resources.  This 
Includes  identifying  the  areas  of  uncertainty,  assessing  the  probabilities,  analyz¬ 
ing  to  determine  the  driving  or  dominant  parameters,  evaluating  funding  alter¬ 
natives,  making  trade  off  studies,  and  making  decisions  on  course(s)  of  action." 

An  unstable  design  is  "a  design  that  has  a  high  potential  for  change  and  may 
require  more  engineering  in  order  to  meet  the  design  specification  requirements 
for  operational  performance,  producibility ,  maintainability,  or  reliability. 

Design  stability  considerations  include  determination  or  confirmation  of  the 
equipment  failure  mode  and  effects  pattern  under  normal  operating  circumstances." 


INTERVIEW  GUIDE  FOR  DESIGN  STABILITY  STUDY 


1.  Design  instability  is  often  cited  as  a  reason  for  using  interim  contractor 
support  (ICS).  How  would  you  define  design  stability  vs.  instability  as  it 
relates  to  such  a  decision,  and  to  logistics  supportability  of  a  new/modified 
weapon  system? 

.  Do  any  of  the  following  items  have  an  effect  on  the  date  at  which  a  weapon 
stem  achieves  logistics  supportability?  If  so,  please  indicate  whether  the 
effect  is  significant  or  minimal.  Include  any  additional  items  that  may  affect 
logistics  supportability  under  "other".  Please  rank  according  to  impact 
(1  -  most  impact,  etc.).  Ties  are  allowed  if  necessary. 


Priority  Impact 

Sis  Min 


a.  Number  and  timeframe  of  engineering 
change  proposals  (ECPs) . 

b.  Validation  of  logistics  support 
elements. 

c.  Length  of  conceptual  phase. 

d.  Length  of  full  scale  development 
(FSD)  phase. 

e.  Concurrency  of  FSD  and  production  programs. 

f.  Complexity  of  equipment  (i.e.,  leading 
edge  of  technology) 

g.  Initial  operational  capability  (IOC)  date. 

h.  First  article  delivery  date. 

i.  Support  equipment  development. 

j.  Critical  design  review  (CDR)  date 

k.  Functional  configuration  audit  (FCA)  date. 

l.  Physical  configuration  audit  (PCA)  date. 

m.  Other  (please  specify) 


3.  What  is  the  effect  on  the  following  of  a  delay  in  reaching  design  stability? 
Indicate  significant,  minimal,  or  none. 


1-3  month  slide  3+  month  slide 

sig  min  none  sig  min  none 

a.  Support  equipment  i iquirement 

document  (SERD)  development  _  _  _  _  _  _ 

b.  Support  equipment  development  _  _  _  _  _  _ 

c.  Support  equipment  delivery  _  _  _  _  _  _ 

d.  Technical  document  preparation  _  _  _  _  _  _ 

e.  Interim  contractor  support  (ICS)  _  _  _  _  _  _ 

f.  Logistics  support  analysis  (LSA)  _  _  _  _  _  _ 

g.  Training  equipment  development  _  _  _  _  _  _ 

h.  Training  development  _  _  _  _  _  _ 

i.  Spares  computations  (provisioning)  _  _  _  _  _  _ 

J.  Spares  delivery  dates  _  _  _  _  _  _ 

k.  Organic  capability  date  _  _  _  _  _  _ 

l.  Life  cycle  costs  (LCC)  _  _  _  _  _  _ 

m.  Other  (please  specify) 


A.  One  technique  currently  used  to  assess  design  stability  is  a  trend  analysis 
of  engineering  change  proposals  (ECPs) .  Please  identify  any  other  techniques 
which  you  have  used  'or  are  aware  of)  for  assessing  the  design  stability  of  a 
weapon  system  from  a  logistics  supportability  viewpoint. 

5.  Prior  to  making  a  decision  to  use  ICS,  AFR  800-21  requires  "rigorous  cost 
and  risk  analyses"  to  support  such  a  decision.  Can  you  comment  on  the  adequacy 
of  techniques  available  to  perform  such  analyses? 

6.  What  techniques  have  you  employed  in  the  past  to  perform  such  analyses? 

7.  Please  list  all  such  techniques  with  which  you  are  familiar  and  rank  them 
according  to  how  effective  you  believe  them  to  be.  (I  *  most  effective,  etc.) 


8.  The  previous  seven  questions  have  addressed  the  point  of  design  stability  vs. 
instability  as  it  relates  to  decisions  about  the  use  of  ICS.  Would  you  care  to 
comment  on  the  feasibility  of  establishing  such  a  point,  and  on  its  usefulness 
in  making  such  decisions? 

9.  For  programs  with  which  you  are  familiar,  which  of  the  two  situations  described 
in  AFR  800-21  resulted  in  a  requirement  for  ICS?  (Please  include  all  programs 
with  which  you  are  familiar.) 

10.  Do  you  have  any  additional  comments  on  this  subject? 


APPENDIX  C 
INTERVIEW  FORMAT 


1?D-Al35  008  AN  ANALYSIS  OF  THE  FEASIBILITY  OF  USING  DESIGN  2/2 

STABILITY  AS  A  DECISION  PA.  .  <U>  AIR  FORCE  INST  OF  TECH 
URIGHT-PATTERSON  AFB  OH  SCHOOL  OF  5VST.  .  D  E  GOOD 
UNCLASSIFIED  SEP  82  AFIT-LSSR-84-82  F/G  15/5  NL 


MICROCOPY  RESOLUTION  TEST  CHART 

NATIONAL  BUREAU  OF  STAN DARDS  - 1 963-A 


1.  How  many  years  of  logistics  management  experience 
do  you  have? 

2.  How  many  years  of  experience  do  you  have 
involving  the  management  of  ICS? 

3.  How  many  ICS  programs  have  you  been  involved 
with? 

4.  What  types  of  weapon  systems  were  they? 

(Please  list  all  types.) 

5.  Now,  I'd  like  to  simply  follow  the  interview 
guide  worksheet  and  discuss  your  responses  to 
those  questions.  I'd  like  to  keep  your  completed 
worksheet  when  we  are  through.  If  you  have  any 
questions  about  the  worksheet,  please  ask  them  as 
we  get  to  the  appropriate  point. 

(Begin  worksheet  questions) 

6.  What  is  your  response  to  question  number  _ 

on  the  worksheet? 
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APPENDIX  D 

ECP  TREND  ANALYSIS 
COMPUTER  DIAGNOSTICS 
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PftHHITEHED  F15  ECP  DATA 
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F-15  FINAL  MODEL 
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OBSERVED  SERIES  PACE 
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F-15  FINAL  MODEL 
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f-15  FINAL  MODEL 
PRENHITENED  FIS  ECP  BATA 
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F-16  FINAL  MODEL 
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No  specific  milestone 


Do  any  of  the  following  items  have  an  effect  on  the  date  at  which  a  weapon 
system  achieves  logistics  supportability?  If  so,  please  indicate  whether  the 
effect  is  significant  or  minimal.  Include  any  additional  items  that  may  effect 
logistics  supportability  under  "other.-  Please  rank  according  to  impact  (1  = 
most  impact,  etc.).  Ties  are  allowed  if  necessary. 
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What  is  the  effect  on  the  following  of  a  delay  in  reaching  design  stability? 
Indicate  significant,  minimal,  or  none. 
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One  subject  gave  separate  responses  for  intermediate  and  depot  level  impacts. 
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